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FOREWORD 


This  guidebook  was  prepared  as  part  of  the  Software  Acquisition  Engineering 
Guidebooks  contract,  F33657-76-C-0723.  It  describes  the  status  measurement  and 
reporting  associated  with  Air  Force/Contractor  software  procurement  as  applied  to 
Training  Simulators  and  Automatic  Test  Equipment.  It  is  primarily  intended  for  use 
by  USAF  acquisition  engineering  personnel. 

This  guidebook  is  one  of  a  series  intended  to  assist  the  Air  Force  Program  Office 
and  engineering  personnel  in  software  acquisition  engineering  for  automatic  test 
equipment  and  training  simulators.  Titles  of  other  guidebooks  in  the  series  are 
listed  in  the  introduction.  These  guidebooks  will  be  revised  periodically  to 
reflect  changes  in  software  acquisition  policies  and  feedback  from  users. 

This  guidebook  reflects  an  interpretation  of  DOD  directives,  regulations  and 
specifications  which  were  current  at  the  time  of  guidebook  authorship.  Since 
subsequent  changes  to  the  command  media  may  invalidate  such  interpretations  the 
reader  should  also  consult  applicable  government  documents  representing  authorized 
software  acquisition  engineering  processes. 

This  guidebook  contains  alternative  recommendations  concerning  methods  for 
cost-effective  software  acquisition.  The  intent  is  that  the  reader  determine  the 
degree  of  applicability  of  any  alternative  based  on  specific  requirements  of  the 
software  acquisition  with  which  he  is  concerned.  Hence  the  guidebook  should  only  be 
implemented  as  advisory  rather  than  as  mandatory  or  directive  in  nature. 

This  guidebook  was  prepared  by  the  Boeing  Aerospace  Company. 


This  Software  Acquisition  Engineering  Guidebook  is  one  of  a  series 
prepared  for  Aeronautical  Systems  Division,  Air  Force  Systems  Command, 
Wright-Patterson  AFB  OH  45433.  Inquiries  regarding  guidebook  content 
should  be  sent  to  ASD/ENE,  Wright-Patterson  AFB  OH  45433.  The  following 
list  presents  the  technical  report  numbers  and  titles  of  the  entire 
Software  Acquisition  Engineering  Guidebook  Series.  Additional  copies  of 
this  guidebook  or  any  other  in  the  series  may  be  ordered  from  the  Defense 
Documentation  Center,  Cameron  Station,  Alexandria  VA  22314. 
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Section  1.0  INTRODUCTION 


Numerous  DOD  studies  and  publications 
have  focused  on  the  importance  of  soft¬ 
ware  management  visibility  early  in  pro¬ 
ject  development  stages.  Although  pro¬ 
jects  are  staffed  with  competent 
personnel,  planning  and  organization  may 
be  lacking  such  that  problems  can 
develop,  spread  and  seriously  impact  a 
project.  Techniques  for  early  problem 
recognition,  planned  abatement  methods 
and  follow-up  are  necessary  to  resolve 
both  predictable  and  unforeseen  software 
development  difficulties.  This  guide 
book  presents  methods  applicable  to 
Automatic  Test  Equipment  (ATE)  and 
Training  Simulator  (TS)  system  software 
acquisition  for  measuring  and  reporting 
software  development  problems  and 
correcting  them  before  they  cause  major 
system  problems. 

1.1  PURPOSE 

The  purpose  of  measuring  and  reporting 
is  to  discover  and  correct  problems 
threatening  the  project.  This  guidebook 
identifies  the  parameters  which  measure 
software  development  progress  and 
describes  methods  of  reporting  and  pre¬ 
senting  that  progress.  It  further 
describes  methods  of  recognizing  and  cor¬ 
recting  software  problems.  It  is  an  aid 
to  USAF  planners  in  imposing  require¬ 
ments  for,  and  participating  in,  contrac¬ 
tor  management  visibility  activities. 

1.2  SCOPE 

This  is  one  of  a  series  of  guidebooks 
related  to  the  Software  Acquisition  Engi¬ 
neering  (SAE)  process  for  TS  and  ATE 
ground  based  systems.  The  SAE  guidebook 
titles  are  listed  below: 

Software  Cost  Measuring  and  Reporting 
Requirements  Specification 
Contracting  for  Software  Acquisition 
Statement  of  Work  (SOW)  and  Requests 
for  Proposal  (RFP) 

Regulations,  Specification  and  Stan¬ 
dards 

Measuring  and  Reporting  Software 
Status 


Computer  Program  Documentation 
Requirements 

Software  Quality  Assurance 
Verification 

Validation  and  Certification 
Computer  Program  Maintenance 
Software  Configuration  Management 
Reviews  and  Audits 

Management  Reporting  by  the  Software 
Di rector 

This  guidebook  covers:  AF  and  contractor 
management  visibility  planning;  factors 
to  conside  in  measuring  software  devel¬ 
opment  progress;  methods  of  reporting 
status;  and  recognizing,  categorizing 
and  correcting  problems  with  emphasis  on 
the  unique  aspects  of  ATE  and  TS  soft¬ 
ware. 

Status  monitoring  methods  discussed  in 
this  guidebook  cover  activities  begin¬ 
ning  with  the  full  scale  development 
phase  (i.e.  analysis,  design,  code  and 
checkout,  test  and  integration,  installa¬ 
tion  and  operation  and  support)  as 
described  in  the  Computer  Program  Devel¬ 
opment  Plan  (CPDP)  and  as  covered  in  the 
guidebook  on  "Software  Cost  Measuring 
and  Reporting." 

1.3  TS  AND  ATE  OVERVIEW 

This  section  provides  a  brief  sketch  of 
TS  and  ATE  system  characteristics, 
including  the  function  of  the  software 
associated  with  each. 

1.3.1  TS  System  Characteristics 

The  TS  system  is  a  combination  of  spe¬ 
cialized  hardware,  computing  equipment, 
and  software  designed  to  provide  a  syn¬ 
thetic  flight  and/or  tactics  environment 
in  which  aircrews  learn,  develop  and 
improve  the  skills  associated  with  indi¬ 
vidual  and  coordinated  tasks  in  specific 
mission  situations.  Visual,  aural,  and/ 
or  motion  systems  may  be  included.  Fig¬ 
ure  1.3-1  depicts  a  typical  TS  which 
employs  digital  processing  capability. 
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The  computer  system,  integral  to  the  new 
TS  can  consist  of  one  or  more  general 
purpose  computers.  The  computing 
hardware  operates  with  floating  point 
arithmetic  and  sufficient  bit  capacity 
to  provide  efficient  use  of  a  simulator 
Higher  Order  Language  (HOL). 

When  a  multi-processor/multi-computer 
system  is  used,  it  must  be  designed  such 
that  computers  can  operate  simulta¬ 
neously  and  are  controlled/synchronized 
by  a  single  program  (supervisor/execu¬ 
tive).  The  executive  directs  program 
execution  and  regulates  priorities.  The 
simulator  (1)  accepts  control  inputs 
from  the  trainee  (via  crew  station  con¬ 
trols)  or  from  the  instructor  operator 

station;  (2)  performs  a  realtime  solu¬ 
tion  of  the  simulator  mathematical 

model;  and  (3)  provides  output  responses 
necessary  to  accurately  represent  the 
static  and  dynamic  behavior  of  the  real 
world  system  (within  specified  tolerance 
and  performance  criteria). 

Since  TS  consist  of  interdependent 
hardware  and  software,  a  joint 

hardware/software  development  effort  is 
required.  As  the  complexity  of  TS 
increase,  simulation  software  continues 
to  grow  in  complexity,  size,  and  cost. 
Software  costs  can  and  do  exceed 

computer  hardware  costs  in  many  cases. 
Therefore,  it  is  imperative  that  the 
simulation  software  acquisition 
engineering  process  be  subjected  to  for¬ 
mal  system  engineering  planning  and  dis¬ 
cipline  to  ensure  cost-effective  procure¬ 
ment. 

1.3.2  ATE  System  Characteristics 

ATE  is  defined  as  that  ground  support 
system  which  performs  vigorous  system 
tests  with  minimum  manual  intervention. 
ATE  is  used  in  place  of  manual  devices 
because  it  is  more  cost  effective, 
provides  required  repeatability,  or 
repair  of  the  item  being  tested  requires 
the  speed  which  only  an  automatic  tester 
can  achieve.  Figure  1.3-2  shows  the 
typical  components  of  an  ATE  system. 


Note  that  there  are  both  hardware  and 
software  elements  involved.  Most  of  the 
elements  shown  in  the  figure  will  be 
found  in  the  majority  of  ATE  systems 
although  the  packaging  and  interface 
design  may  vary  between  specific 
systems. 

The  controls  and  displays  section  con¬ 
sists  of  the  computer  peripheral  devices 
such  as  control  panels,  magnetic  tape 
cassettes  or  disks,  a  cathode  ray  tube 
(CRT),  keyboard.,  and  small  printer.  The 
computer  (normally  a  minicomputer),  as 
controlled  by  software,  operates  the 
peripheral  devices;  switches  test  stim¬ 
uli  on  and  off;  and  measures  responses 
of  the  Unit  Under  Test  (UUT)  (comparing 
to  predetermined  values).  The  operator 
maintains  supervisory  control  of  the 
testing  process  through  the  peripherals. 
However,  his  interaction  is  usually  mini¬ 
mal  since,  by  definition,  the  automatic 
test  feature  was  selected  in  preference 
to  an  operator-controlled  test  system. 

ATE  is  normally  designed  to  accomodate 
testing  several  different  articles  of 
system  equipment  (normally  one  at  a 
time).  The  maintenance  level  being  sup¬ 
ported  by  the  ATE  is  determined  by  logis¬ 
tics  systems  analysis.  The  importance  of 
the  software  portion  of  the  ATE  system 
should  not  be  minimized  since  both  the 
application  of  the  test  stimuli  and  the 
measurement  of  the  result  are  achieved 
via  software.  Arbitrary  function  genera¬ 
tion  and  complicated  wave  analysis  can 
also  be  accomplished  by  software  and  is 
becoming  more  prevalent  in  ATE  systems. 
The  cost  of  ATE  software  is  a  signifi¬ 
cant  component  of  total  ATE  costs  and 
design  trades  can  be  performed  to  mini¬ 
mize  ATE  life-cycle  costs. 

1.4  GUIDEBOOK  ORGANIZATION 

The  guidebook  is  organized  as  follows. 
Section  1.0  is  introductory.  Section  2.0 
identifies  government  and  industry  publi¬ 
cations  and  data  items  dealing  with  man¬ 
agement  visibility  for  software  develop¬ 
ment.  Section  3.0  discusses  planning  for 


3 


hardware  and  software  phasing  and  correcting  problems.  Section  6.0 
tracking  adherence  to  their  respective  discusses  unique  ATE  and  TS 
development  milestones.  Section  4.0  considerations  relevant  to  measuring  and 
discusses  actual  status  measuring  and  reporting  software  status.  Sections  7.0 
reporting  mechanisms.  Section  5.0  through  11.0  contain  a  bibliography, 
identifies  methods  of  recognizing  and  guidebook  topic  vs  DOD  document 

cross-reference  index,  glossary,  list  of 
abbreviations  and  acronyms,  and  a 
subject  index. 
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Section  2.0  APPLICABLE  DOCUMENTS 


Government  documents  dealing  with  the  AFR  800-2,  Program  Management 
topics  covered  by  this  guidebook  are: 

AFR  800-14,  Management  of  Computer 
MIL-STD  483,  Configuration  Management  Resources  in  Systems 
Practices  for  Systems  Equipment,  Muni¬ 
tions  and  Computer  Programs.  SSD  Exhibit  61-47B,  Computer  Program 

Subsystem  Development  Milestones 

MIL-STD  1521A,  Technical  Reviews  and 
Audits  for  Systems,  Equipment  and  Com¬ 
puter  Programs. 
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Section  3.0  HARDWARE  AND  SOFTWARE  PHASING- MANAGEMENT  VISIBILITY 


In  a  ground  system  such  as  ATE,  opera¬ 
ting  system  software  is  a  part  of  the 
system  "central  core"  which  in  turn  is 
linked  with  a  functional  test  station, 
an  Interface  Test  Adapter  (ITA)  and  a 
UUT  test  program  to  comprise  a  total 
system.  Since  the  development  of  these 
components  is  inter-dependent  (i.e., 
software  design  affecting  hardware 
design  in  hardware),  decisions  are 
required  to  allocate  requirements  and 
prioritize  developmental  work.  This 
section  provides  guidelines  for  software 
scheduling,  prioritizing  and  status 
monitoring. 

3.1  ESTABLISHING  PROJECT  MILESTONES 

Schedules  are  time  phased  representa¬ 
tions  of  a  plan  often  displayed  in  graph¬ 
ic  form.  They  are  essential  to  the 
successful  acquisition  of  ATE  and  TS 
since  they  represent  one  of  the  founda¬ 
tions  of  effective  management.  Experi¬ 
ence  has  shown  the  real  key  to  effective 
software  management  involves  three  essen¬ 
tial  elements.  First,  it  requires  devel¬ 
opment  of  a  credible  plan.  The  schedule 
is  of  course  fundamental  to  this  pro¬ 
cess.  Second,  a  system  of  measurements 
and  reports  is  established  to  determine 
actual  performance  against  the  plan. 
Finally  corrective  action  is  taken  when 
actual  performance  deviates  from  the 
plan.  Schedules  and  their  effective  con¬ 
trol  are  helpful  to  the  TS  and  ATE 
acquisition  engineer  for  the  following 
reasons. 

3.1.1  Milestone  Accomplishment 

Milestones  provide  a  greater  degree  of 
reliability  that  the  system  will  be 
delivered  on  time.  Critical  milestones 
are  established.  These  milestones 
reflects  events  which  occur  during  the 
life  of  the  system  acquisition.  Because 
the  acquisition  engineer  must  monitor 
and  evaluate  contractor  performance  and 
must  be  aware  when  his  actual 
performance  deviates  from  the  schedule, 
these  milestones  should  be  selected  with 
care.  By  this  is  meant  the  following: 


a.  Each  milestone  event  should  be 
unambiguous  in  nature.  Schedule  mile¬ 
stones,  particularly  those  on  which  con¬ 
tractual  matters  are  based,  are  a  sub¬ 
ject  of  negotiation  during  the  fact 
finding  and  contract  definitization 
phase  of  the  source  selection  process. 
This  process  is  described  in  the  Con¬ 
tracting  for  Software  Acquisition  Guide¬ 
book.  During  negotiation  of  these 
milestones  particular  attention  should 
be  placed  on  ensuring  the  milestone 
event  can  be  recognized  as  such  when  it 
has  occurred.  For  example,  a  milestone 
event  described  as  delivery  of  the 
software  part  II  specifications  leaves 
little  doubt  as  to  what  is  meant.  The 
term  "delivery"  implies  a  contractually 
prescribed  sequence  of  events  which  must 
take  place.  Little  if  any  doubt  exists 
in  determining  when  these  events  have 
occurred.  However,  a  milestone  described 
as  completion  of  software  part  II 
specifications  can  be  ambiguous.  The 
term  "complete"  can  be  defined  in  a 
number  of  different  ways.  It  can  mean 
the  specifications  have  been  written, 
printed  and  are  waiting  for  company 
management  to  review  them.  They  may  not 
be  available  to  the  USAF  for  sometime  to 
come.  Or,  it  can  mean  the  contractor 
understands  and  has  defined  the 
specifications  but,  they  have  not  been 
written,  reviewed  or  made  available  to 
the  USAF. 

b.  Milestones  should  describe  tangi¬ 
ble  events  wherever  possible.  A  mile¬ 
stone  stating  "analysis  complete"  is  not 
a  tangible  event.  It  is  often  difficult 
to  determine  when  this  event  has 
occurred.  However,  if  a  milestone  such 
as  "analysis  document"  is  used,  a  more 
tangible  event  has  been  described.  In 
the  latter  case  a  specific  product  is 
produced  which  can  be  evaluated.  In  the 
former  case  only  a  concept,  or  the  state 
of  mine  of  the  analyst  is  described  and 
it  is  difficult  to  determine  the  precise 
date  that  such  an  event  occurs.  In 
general,  milestones  should  be  selected 
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as  those  points  in  time  at  which  a  spe¬ 
cific  tangible  product  such  as  a  soft¬ 
ware  document  or  test  report  has  been 
produced. 

c.  Milestones  should  occur  at  signi¬ 
ficant  events.  The  purpose  of  a  mile¬ 
stone  event  is  to  measure  partial  comple¬ 
tion  at  an  intermediate  step  and  use 
this  as  a  measure  of  the  degree  to  which 
the  entire  job  is  complete.  Therefore, 
milestones  of  events  which  are  necessary 
preliminary  steps  in  completion  of  the 
entire  software  system  should  be 
selected.  If  instead,  insigificant 
events  or  events  which  are  not  in  the 
main  stream  of  software  development  are 
chosen,  the  purpose  for  milestone  sched¬ 
uling  is  compromised.  For  example,  the 
Program  Management  Plan  ( PMP ) ,  Computer 
Resources  Integrated  Support  Plan 
(CRISP),  and  Test  Requirements  Document 
(TRD)  are  significant  events  in  the 
development  of  ATE  and  TS  software.  Com¬ 
pletion  of  these  activities  is  a  neces¬ 
sary  step  which  precedes  coding  and 
checkout  of  the  software.  Consequently, 
availability  of  satisfactory  versions  of 
these  provide  an  indication  of  the 
degree  to  which  the  entire  software  job 
is  complete.  On  the  other  hand  a  mile¬ 
stone  established  at  the  completion  of  a 
software  subroutine  may  not  be  indica¬ 
tive  of  the  degree  of  total  job  comple¬ 
tion  since  the  subroutine  could  be  devel¬ 
oped  independently. 

d.  Milestones  should  be  placed  at 
intervals  which  roughly  coincide  with 
the  intervals  of  status  reporting.  To 
provide  an  excessive  number  of  sequen¬ 
tial  milestones  all  falling  due  in  the 
same  reporting  period  is  meaningless. 
Further,  it  can  waste  contractor  and 
government  effort  required  to  define, 
measure,  report  and  review  actual  per¬ 
formance  against  these  milestones.  If 
too  few  milestones  are  scheduled,  there 
will  exist  risk  that  problems  will  go 
unreported  and  both  the  contractor  man¬ 
agement  and  the  government  may  lose  con¬ 
trol  of  the  software  job. 

e.  The  majority  of  ATE  and  TS  pro¬ 
jects  involve  software  of  sufficient 


scope  to  require  a  hierarchy,  or  multi¬ 
ple  tiers  or  schedules.  Milestones  for 
these  should  be  identified  so  as  to 
achieve  consistency  throughout  each  tier 
of  a  multi-tiered  schedule.  Each  succeed¬ 
ing  tier  after  the  highest  level  divides 
the  same  software  job  into  finer  degrees 
of  scheduling  and  reporting  detail. 
Therefore,  continuity  should  be  main¬ 
tained  so  that  it  is  possible  to  locate 
every  higher  tiered  milestone  on  each 
succeeding  lower  tiered  schedule. 
Further  consistency  in  naming  of  events 
should  be  maintained  so  that  lower  tier 
milestone  could,  if  desired,  be  located 
with  respect  to  any  higher  tiered  sched¬ 
ule.  Care  should  be  maintained  to  convey 
the  message  that  each  lower  tiered  sched¬ 
ule  is  a  more  detailed  plan  of  the  same 
job  reflected  on  a  higher  tier  schedule 
rather  than  appearing  as  a  schedule  for 
a  totally  different  job. 

f.  The  establishment  of  project  mile¬ 
stones  makes  it  possible  to  accomplish 
tasks  in  a  more  orderly  fashion.  The 

very  discipline  required  to  think 
through  a  complex  TS  or  ATE  software 
job,  and  define  events  for  purposes  of 
milestone  establishment  and  associated 
flow  times  of  activities  leading  to 
these  events  is  in  itself  an  extremely 

useful  practice.  To  do  this  requires 
definition  of  every  principal  software 
element,  every  major  document,  every 
significant  portion  of  data,  and  every 
algorithm  by  which  the  input  is  trans¬ 
lated  to  the  output.  In  short,  to  do 

this  scheduling  process  requires  the  con¬ 
tractor  software  manager  and  his 
acquisition  engineering  counterparts  in 
the  USAF  to  define  all  the  software 

performance  and  documentation.  This  is 
the  first  necessary  step  in  any  suc¬ 
cessful  software  development  activity. 


Schedule  control  is  a  primary  means  of 
determining  potential  problems  in  time 
to  institute  effective  corrective 
action.  Methods  of  recognizing  and 
abating  such  problems  are  described  in 
Section  5.0.  Schedule  control  is  an 
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essential  element  in  estimating,  allo¬ 
cating  and  planning  manpower  and  other 
required  resources.  Government  contracts 
today  are  requiring  ever  increasing 
levels  of  detail  in  the  evaluation  of 
contractor  proposals  and  in  fact 
finding.  In  software  cost  estimating, 
the  primary  resource  is  manpower.  Man¬ 
power  estimates,  in  turn,  are  derived 
from  master  and  lower  tiered  schedules. 
Software  resource  planners  must  divide 
the  overall  task  into  time  phased  sub¬ 
tasks  compatible  with  the  master  sched¬ 
ule  and  contractual  milestones.  Unreal¬ 
istic  scheduling  can  seriously  impact 
the  success  of  the  software  development 
effort;  particularly,  if  the  required 
resources  (i.e.  special  skills)  are  not 
available  when  they  are  needed. 

3.2  DEFINING  MILESTONE  PRODUCTS 

In  any  system  acquisition,  the  contract 
dictates  the  Tier  1  milestones  which 
form  the  "backbone"  of  all  subtier  sched¬ 
ules.  So  called  "Tier  1  milestones" 
usually  include,  but  are  not  limited  to, 
"contractual  milestones."  Contractual 
milestones  for  software  development  are 
twofold;  events  and  deliveries/ 
submittals.  Some  typical  contractual 
milestones  for  ATE  and  TS  software 
acquisition  and  development  are: 

a.  Events 

(1)  System  Requirements  Review 
(SRR) 

(2)  System  Design  Review  (SDR) 

(3)  Preliminary  Design  Review 
(PDR) 

(4)  Critical  Design  Review  (CDR) 

(5)  Functional  Configuration  Audit 
(FCA) 

(6)  Physical  Configuration  Audit 
( PCA ) 

(7)  Formal  Qualification  Review 
(FQR) 


b.  Deliveries/Submittals 

(1)  Prime  Item  Development  Spec¬ 
ifications  (PIDS) 

(2)  Configuration  Item 
(Cl) /Computer  Program 
Configuration  Item  (CPCI) 
Specifications 

(3)  Computer  Program  Development 
Plan  (CPDP) 

(4)  Configuration  Management  Plan 
(CMP) 

(5)  Interface  Control  Drawing 
(ICD) 

(6)  Test  Requirements  Documents 
(TRD) 

(7)  Qualification  Test  Procedures 

(8)  Acceptance  Test  Procedures 

(9)  Qualification  Test  Report 

(10)  Acceptance  Test  Reports 

(11)  User's  Manual 

(12)  Version  Description  Document 
(VDD) 

(13)  Computer  Program  Media 

(14)  System  Delivery 

Beginning  with  these  basic  contractual 
milestones,  schedulers  should  prepare 
master  and  subtier  schedules.  It  is 
recommended  that  a  separate  contractor 
organization  exist  to  prepare  and  main¬ 
tain  these  very  important  schedules. 
Usually  the  contractor  organization  per¬ 
forming  this  function  is  Program 
Planning  and  Control  ( PP&C ) .  Scheduling 
is  a  skill  that  requires  a  certain 
aptitude  and  specialized  training.  The 
critical  qual if i cat  ion  for  a  good 
scheduler  is  common  sense.  Some  highly 
competent  engineers  and  design  managers 
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are  not  good  schedulers.  They  are 
frequently  so  preoccupied  with  finding 
solutions  to  troublesome  design 
problems,  that  they  do  not  "see  the 
forest  through  the  trees." 

3.2.1  Scheduling 

Effective  scheduling  demands  a  dedicated 
and  sustained  effort  and  should  be  per¬ 
formed  by  a  group  skilled  in  schedule 
preparation  and  maintenance.  This  group 
is  responsible  for  the  following  tasks. 

a.  Identification  of  all  required 
tasks  and  interdependencies  that  exist 
between  tasks. 

b.  Accomplishment  of  tasks  and  deliv¬ 
ery  of  output  products  in  an  appropiate 
and  logical  sequence. 

c.  Establishment  of  an  accountability 
system  for  monitoring  completion  of  all 
required  tasks  and  delivery  of  all 
required  output  products. 


These  resources  of  necessity  cross  organ¬ 
ization  boundaries  and  must  be  under  the 
control  of  the  scheduling  organization. 
It  is  reemphasized  that  milestone  defini¬ 
tions  should  be  tangible  entites.  Func¬ 
tional  organizations  contributing  to  the 
overall  schedule  objectives  should 
clearly  understand  what  is  needed  at  a 
given  point  in  time  and  the  interfacing 
organizations  using  a  given  organiza¬ 
tion's  input  must  clearly  communicate 
what  is  needed. 

3.2.3  Communications 

The  establishment  and  maintenance  of 
these  needed  communications  is  estab¬ 
lished  as  follows: 

a.  Schedule  Planning  and  "Networking" 

b.  Schedule  development 

c.  Schedule  Statusing 

d.  Management  Review 


d.  Anticipation  of  resource  require¬ 
ments  -  neither  too  little,  too  late, 
nor  too  much  too  soon,  but  what  is 
needed,  when  it  is  needed,  and  where  it 
is  needed. 

3.2.2  Resources 

Resources  which  must  be  anticipated, 
quantified,  scheduled  and  provided  for 
are: 


e.  Problem  Identification  and  Abate¬ 
ment 

The  following  subsections  describe  each 
of  the  above  major  steps  in  schedule 
development  and  monitoring  in  synoptic 
fashion.  Subsequent  sections  of  the 
guidebook  elaborate  on  the  detailed 
techniques  of  schedule  development  and 
monitoring  including  considerations  uni¬ 
que  to  ATE  and  TS  software. 


a.  Time  -  manhours/manmonths  of 
effort  and  the  arrangement  (prioritiza¬ 
tion)  of  that  effort. 

b.  Skills  -  having  the  required  per¬ 
sonnel  with  the  requisite  skills  avail¬ 
able  when  needed. 

c.  Materials  8  Facilities  -  ensuring 
the  availability  of  adequate  program 
development  facilities  and  peripheral 
and  the  communications  network  available 
with  the  required  capabilities  and  capa¬ 
cities. 


3.3  SCHEDULE  DEVELOPMENT  AND 
MAINTENANCE 

An  effective  schedule  status  function, 
properly  staffed  and  fully  supported  by 
management  will  not  necessarily  prevent 
a  project  from  experiencing  schedule  pro¬ 
blems.  However,  the  advantage  of  such 
systems  is  that  potential  problem  areas 
are  highlighted  at  an  early  stage  when 
effective  corrective  action  can  be  imple¬ 
mented.  An  effective  schedule/status  dis¬ 
cipline  is  difficult  to  implement  for 
software  development  projects  because  it 
takes  time,  dedicated  resources  and 
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reporting  commitments  from  personnel 
unaccustomed  to  this  type  of  discipline. 
However,  if  goals  are  realistic  (and 
time  is  allocated  to  programmers  to  pro¬ 
vide  schedule  information),  the  project 
manager  will  find  the  process  well  worth 
the  effort.  The  following  are  the  basic 
steps  in  estabishing  and  maintaining 
software  schedules. 

3.3.1  Schedule  Planning  and  Networking 

Starting  with  need  dates  and  backing  up 
from  there,  the  first  step  the  software 
project  manager  must  undertake  is  the 
planning  of  a  schedule  or  building  a  net¬ 
work.  The  "network"  concept  is  a  method 
for  representing  a  logically  sequenced 
schedule  plan  in  graphic  form.  The  steps 
necessary  in  developing  a  network  are: 

a.  Develop  a  statement  of  work  (SOW) 
of  what  must  be  done. 

b.  Identify  tasks,  activities,  out¬ 
puts  required  to  achieve  the  end  objec¬ 
tive. 

c.  Relate  interdependencies  of  tasks. 

d.  Assign  responsibilities. 

e.  Assign  estimated  completion  dates 
(ECD's). 

Figure  3.3-1  illustrates  a  simplified 
development  network  for  a  given  CPC I .  A 
complete  network  would  contain  an  organi¬ 
zational  or  personnel  assignment  for 
each  activity  or  task.  At  this  point,  no 
mention  will  have  been  made  of  the 
amount  of  time  necessary  to  accomplish 
each  activity,  since  at  this  point  the 
scheduler  is  only  concerned  with 
defining  and  sequencing  events.  Next  the 
project  manager  must  allocate  the  avail¬ 
able  time  between  the  present  date  and 
the  final  job  completion  date  into  inter¬ 
vals  appropriate  for  each  subtask.  Work 
normally  performed  in  a  serial  fashion 
may  have  to  be  done  in  parallel.  Next, 
the  interdependencies  of  these  tasks 
must  be  evaluated.  For  example,  the  ques¬ 
tion,  "do  modules  A  and  B  really  have  to 
be  done  prior  to  beginning  design  at 


module  C?"  must  be  answered.  Perhaps 
schedule  difficulties  can  be  partially 
alleviated  by  parallel  development 
efforts  or  possibly  some  of  the  design 
effort  may  be  consolidated.  Finally, 
upon  completing  a  realistic  planning  net¬ 
work,  the  project  manager  should  assign 
personnel  responsibilities  to  each  acti¬ 
vity  to  insure  adequacy  and  quantity  of 
skills  to  accomplish  this  preliminary 
plan.  The  project  manager  should  review 
this  plan  with  his  staff  to  insure  a 
clear  understanding  of  each  programmer's 
responsibilities.  Wherever  possible  the 
outputs  of  each  plan  element  should  be 
tangibly  defined.  The  graphical  presenta¬ 
tion  of  the  charted  plan  should,  for  sim¬ 
plicity,  contain  only  a  brief  but 
succinct  statement  descriptive  of  that 
activity.  If  further  explanation  is 
required,  each  applicable  event  may  be 
flag-noted  and  appropriate  job  descrip¬ 
tions  can  be  written  on  a  separate 
chart.  Since  the  network  is  the  "blue¬ 
print"  for  the  formal  schedule,  it  is 
extremely  important  to  review  and  obtain 
commitments  from  those  parties  responsi¬ 
ble  for  executing  each  acti vity/task  to 
insure  that  interdependencies  are  cor¬ 
rect  and  manpower  allocations  are  realis¬ 
tic. 

3.3.2  Schedule  Development 

This  activity  involves  translation  of 
the  network  into  a  formal  schedule.  The 
network  is  usually  prepared  by  the  pro¬ 
ject  manager  using  a  "window"  of  time 
available  to  do  the  job  to  meet  a 
required  completion  date.  Time  network 
is  translated  into  various  tiers  of 
schedules  by  the  scheduling  (PP&C) 
organization.  The  actual  resultant  sched¬ 
ule  is  an  annotated  chart  depicting 
start  and  completion  dates  for  signifi¬ 
cant  events  for  each  development  task. 
It  depicts  parallel  and  serial  activity 
annotated  with  flag-notes  and  legends 
which  are  used  as  warnings  to  management 
reviewers  of  actual  or  potential  impact 
of  schedule  slides  or  missed  milestones. 
Figure  3.3-2  depicts  two  tiers  of  simpli¬ 
fied  scheduling  for  a  typical  ATE 
system/software  development  task.  It  is 
seen  how  increasingly  greater  detail  of 


Figure  3.3-1.  Software  Development  Network  -  CPC 1 XYZ 
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required  events  is  exposed  in  the  Tier 
II  schedule  for  the  general  tasks  shown 
in  the  Tier  I.  Several  points  should  be 
considered  in  associating  milestones 
with  schedule  tier  levels: 

a.  A  milestone  representing  a  soft¬ 
ware  task  generally  fall  into  one  of 
four  tier  levels  based  on  the  signifi¬ 
cance  of  the  event  scheduled  and  the 
level  of  detail  associated  with  the 
respective  tier. 

b.  The  process  of  scheduling  mile¬ 
stones  is  best  performed  from  the  top 
down  (similar  to  the  top-down  design  pro¬ 
cess),  from  high-level  project  mile¬ 
stones  to  low-level  detailed  milestones. 

c.  Scheduling  from  the  top  down  faci¬ 
litates  the  task  of  ensuring  that  sched¬ 
ule  dates  for  detailed  milestones  are 
consistent  and  compatible  with  higher 
level  commitments.  This  becomes  an 
increasingly  difficult  task  as  project 
size  and  complexity  increase. 

Management  schedule  review  systems  for 
controlling  development  of  the  entire 
ATE  or  TS  system  will  typically  employ 
four  tier  levels  of  schedule  monitoring 
as  follows: 

a.  Tier  I  -  contract  and/or  major  pro¬ 
gram  mi Testones  of  the  type  that  would 
appear  on  a  master  schedule.  Examples 
could  be  "requirements  definition  com¬ 
plete,"  "detailed  design  complete,"  or 
"system  installation  complete." 

b.  Tier  II  -  USAF  interface  mile¬ 
stones  that  represent  the  transmittal  of 
major  output  products  either  to  or  from 
the  USAF,  the  completion  of  major 
reviews,  or  the  approval  of  budget  or 
schedule  for  a  succeeding  project  phase. 

c.  Tier  III  -  system/subsystem  inte- 
gration  and  test  milestones  that  gener¬ 
ally  represent  the  completion  of  activi¬ 
ties  above  the  detailed  computer-module 
level  but  below  the  system  level. 


(1)  One  example  is  the  functional 
subsystem  test  of  a  group  of  related  com¬ 
puter  modules. 

(2)  Tier  III  milestones  may  repre¬ 
sent  critical  events  constraining  the 
development  task  but  not  directly  part 
of  it.  Examples  are  conversion  of  data 
from  old  systems,  restructuring  of  old 
data  bases,  implementation  of  new  soft¬ 
ware  or  hardware  critical  to  the  pro¬ 
ject,  implementation  of  distributed  pro¬ 
cessing,  and/or  tele-processing  network 
system  additions  or  deletions. 

d.  Tier  IV  -  detailed  milestones  at 
the  individual  computer-module  level  or 
at  the  individual  data  base  design 
level.  Use  of  standard  milestones  is  a 
particularly  effective  method  for 
measuring  progress  toward  completion 
through  a  common  frame  of  reference  for: 

(1)  Individual  computer-module  de¬ 
sign,  code,  and  test. 

(2)  Individual  data  base  and  devel¬ 
opment. 

For  ATE  and  TS  software  a  set  of  tiered 
schedules  should  be  prepared  for  track¬ 
ing  each  CPCI  or  major  software  compo¬ 
nent.  In  the  case  of  ATE  software,  how¬ 
ever,  the  vendor  supplied  portions  of 
the  control ,  support  and  test  programs 
do  not  warrant  such  tracking  unless 
major  modifications  are  required  to  be 
made  by  the  contractor  or  the  vendor. 
When  all  CPCI ' s  are  scheduled  with  appro- 
pi  ate  detailing  through  the  various 
tiers,  the  initial  schedule  development 
job  is  complete. 

The  formality  and  attention  paid  to  main¬ 
taining  these  schedules  on  an  up-to-date 
basis  cannot  be  overemphasized.  Apathy 
toward  schedule  maintenance  and  adher¬ 
ence  can  destroy  management  visibility 
and  control  of  project.  The  degree  of 
formality  in  managing  schedules  is 
dependent  on  the  size,  duration  and  com¬ 
plexity  of  the  project.  Some  of  the 
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schedule  maintenance,  display  and  moni¬ 
toring  techniques  in  use  today  are: 

a.  Project  dedicated  control  rooms 

b.  Schedule  Planning  Displays 

c.  Tiered  Schedule  Displays 

d.  Standard  Milestones 

e.  Milestone  status  reporting 

f.  Late  item  reporting 

g.  Program  Management  Networks 

The  use  of  these  schedule  management 
tools  is  discussed  in  the  following 
paragraphs. 

3.3.3  Schedule  Statusing 

Once  having  developed  a  complete  and 
realistic  schedule  and  coordinated  with 
all  concerned  functional  organizations, 
it  must  be  vigorously  reviewed,  criti¬ 
qued  and  updated.  This  process  called 
schedule  statusing  measures  actual 
accomplishment  against  commitments, 
reveals  potential  schedule  problems  and 
creates  problem  abatement  planning.  Fig. 
3.3-3  describes  some  typical  symbology 
for  depicting  schedule  adherence. 

In  managing  schedules  for  complex  pro¬ 
jects,  contractor  management  will  typi¬ 
cally  employ  various  color  coded  legends 
for  highlighting  impending  schedule 
impact.  For  example  the  solid  portion  of 
a  bar  chart  designating  "activity  com¬ 
plete"  may  be  color  coded  red  if  the 
activity  is  actually  delinquent,  yellow 
if  potentially  delinquent  or  "black"  if 
"on  schedule."  When  properly  annotated 
with  appropriate  symbology,  footnotes, 
color  coding,  etc.,  the  schedule  should 
be  formally  constructed  on  some  large, 
back  lighted  panel  or  projected  display 
so  that  it  may  be  reviewed  and  critiqued 
by  the  project  management.  Usually  a  con¬ 
trol  room  specially  constructed  for  this 
purpose  (referred  to  in  this  guidebook 
as  a  Management  Information  Center  or 
MIC)  is  usee  to  hold  periodic  "Schedule 


Performance  Reviews."  These  meetings  are 
attended  by  all  responsible  project  man¬ 
agers  and  supporting  personnel. 

While  the  master  and  lower  tiered  sched¬ 
ules  alert  management  to  a  potential  pro¬ 
blem,  further  detail  is  required  to  com¬ 
prehend  the  seriousness  of  the  problem 
and  to  pinpoint  where  recovery  effort 
must  be  placed.  The  software  "Worm" 
chart  (see  Fig.  3.3-4)  provides  visi¬ 
bility  into  percent  complete  as  well  as 
providing  management  an  overview  of 
"rate  of  progress."  The  slope  of  the 
curve  hints  at  whether  behind  schedule 
conditions  are  recovering  or  deteriorat¬ 
ing.  Use  of  such  charting  techniques  are 
further  discussed  in  Section  4.0  for 
both  percent  complete  and  technical  per¬ 
formance  progress  reporting. 

3.3.4  Management  Review 

Management  review  is  the  process  by 
which  key  project  managers  evaluate  pro¬ 
ject  progress.  Tracking  status  against 
milestone  commitments  is  a  critical  com¬ 
ponent  of  schedule  control  discipline. 
The  schedule  board  must  remain  fully 
aware  of  milestone  status  in  order  to 
effect  any  work  resequence/reschedule 
actions  necessitated  by  performance  pro¬ 
blems.  The  various  schedule  conditions 
into  which  management  must  categorize 
activity  and  chart  performance  are: 

a.  Estimated  completion  date 

b.  Potential  slippage 

c.  Actual  slippage 

d.  Completed 

e.  Cancelled 

f.  Suspended 

The  key  challenges  in  schedule  manage¬ 
ment  are: 

a.  Realistic  milestone  establishment 

b.  Accurate  reported  status 
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Figure  3.3-3.  Schedule  Symbology 


Figure  3.3-4.  Software  "Worm"  Chart 


c.  Verification  of  "actuals"  through 
published  evidence  of  completion 

Oftentimes,  although  initial  work  task¬ 
ing  estimates  are  derived  in  good  faith 
based  on  available  data,  pressure  to 
meet  those  commitments  is  felt  by  the 
committing  organization.  There  is  a  ten¬ 
dency  to  either  "shortcut"  the  origi¬ 
nally  scheduled  effort  or  delete  sub¬ 
tasks  not  visible  on  the  higher  tiered 
schedules.  The  key  to  verifying  that  a 
milestnoe  is  in  fact  complete  is  in 
defining  clearly  and  completely  the  out¬ 
put  products  that  are  generated  to  sat¬ 
isfy  the  milestone  requirement,  then 
ensuring  that  each  output  product,  when 
completed,  meets  the  established 
criteria.  The  responsible  organization 
cannot  be  expected  to  perform  this  func¬ 
tion  impartially,  since  there  will  be  a 
built-in  bias  in  favor  of  the  quality  of 
the  output  product.  The  recipient  organi¬ 
zation  should  not  be  expected  to  be 
impartial  either,  since  the  bias  will 
tend  to  be  the  reverse,  an  output 
product  may  never  quite  measure  up  to 
what  is  expected. 

An  effective  procedure  is  to  establish 
an  independent  output  product  release 
monitor  to: 

a.  Verify  and  validate  the  quality 
and  completeness  of  output  products. 

b.  Secure  output  product  acceptabil¬ 
ity  signatures  from  responsible  and 
recipient  organizations  as  required. 

c.  Publish  an  official  record  of  the 
output  products  that  have  met  the  estab¬ 
lished  release  criteria. 


d.  Transmit  the  output  products  in  a 
secure  and  timely  manner  to  the  recip¬ 
ient. 

Software  Quality  Assurance  and  Configura¬ 
tion  Management  organizations  play  a 
vital  role  as  "output  product  release 
monitors"  by  ensuring  through  design  sur¬ 
veillance  and  review  activities  that 
quality  specifications  and  product 
descriptions  are  released  to  define  the 
design.  See  guidebook  on  Software 
Quality  Assurance.  Schedule  reviews  by 
management  should  be  conducted  with 
scrupulous  regularity.  Presenters  must 
be  asked  to  point  out  differences  in 
status  from  the  previous  review  and 
evaluate  impact,  if  any.  Responsible 
project  managers  should  review  schedules 
prior  to  formal  presentation  to  higher 
management  to  identify  problem  areas  and 
report  recovery  plans.  Such  recovery 
plans  should  be  simple,  but  complete  and 
formal  and  should  address: 

a.  Milestone  impacted 

b.  Schedule  milestone  date 

c.  Statement  of  Problem  (reason  for 
slide) 

d.  Recovery  action  plan 

e.  Impact  after  implementing  recovery 
plan 

f.  Project  approval  signatures 

Section  4.0  describes  reporting  and 
review  techniques  applicable  to  ATE  and 
TS  software.  Detailed  problem  abatement 
methods  are  discussed  in  Section  5.0. 
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Section  4.0  STATUS  MEASURING  AND  REPORTING 


This  section  summarizes  the  types  of 
indicators  that  are  identifiable  and 
suitable  for  measuring  software  devel¬ 
opment  status.  In  addition,  there  is  a 
discussion  on  various  reporting  methods 
for  comparing  the  status  against  the 
schedule.  These  reports  provide  input  to 
both  contractor  and  Air  Force  management 
for  recognition  of  areas  of  risk  or  con¬ 
cern. 

4.1  STATUS  MEASUREMENT  PARAMETERS 

Excluding  cost  which  is  the  subject  of 
the  Cost  Measuring  and  Reporting  Guide¬ 
book,  there  are  three  basic  parameters 
which  measure  software  status 
percentage  of  completed  code,  technical 
performance  compliance,  and  docu¬ 
mentation  release.  These  measure  how 
much  is  done;  how  well  the  software 
works;  and  how  current  are  the  required 
engineering  releases.  Once  these  para¬ 
meters  are  determined,  they  are  provided 
for  management  review.  The  following  sub¬ 
sections  discuss  how  to  gather,  orga¬ 
nize,  and  present  the  data  generated 
through  these  parameters. 

4.1.1  Percentage  Complete 

In  tracking  completion  percentage,  a 
scheme  should  be  adopted  which  measures 
each  Computer  Program  Component's 
(CPC's)  progress  against  predetermined 
goals  as  committed  by  previously  sub¬ 
mitted  estimates.  Each  CPC  (module,  sub¬ 
routine,  program  unit,  etc.)  consumes  a 
portion  of  each  major  development  effort 
-  preliminary  design,  detail  design, 
code  and  checkout,  and  test  and  integra¬ 
tion.  As  the  schedule  develops,  esti¬ 
mates  of  percentage  of  overall  effort 
that  each  CPC  will  take  are  evaluated 
and  established.  Percentage  of  completed 
code  is  the  ratio  of  completed  lines  of 
code  to  estimated  total  required  lines 
of  code,  expressed  as  a  percentage.  It 
is  used  to  estimate  degree  of  completion 
of  a  coding  task.  These  estimates  are 
used  to  form  the  basis  for  estimating 
current  status  vs.  the  schedule  for  pur¬ 
poses  of  management  review. 


An  example  of  an  effective  CPC  moni¬ 
toring  technique  is  the  unit  development 
folder  (UDF).  The  UDF  concept  is  a  stan¬ 
dard  for  documenting  plans  and  progress 
of  a  CPC.  The  UDF  is  used  by  the  con¬ 
tractor  in  developing  each  CPC.  It  is 
also  applicable  to  TS  and  ATE  software 
development  performed  by  the  USAF  in  an 
operational  environment.  The  programmers 
are  the  most  common  users  of  the  UDF 
with  the  software/ATE  or  TS  Program  Man¬ 
ager  reviewing  them  at  prescribed  inter¬ 
vals.  The  USAF  acquisition  engineer 
should  specify  in  the  contract  a  proce¬ 
dure  similar  to  the  UDF.  A  large  project 
will  require  a  very  detailed  breakdown 
of  each  CPC  I  into  many  CPCs,  each  with  a 
UDF,  for  development  or  modification.  It 
provides  the  best  available  information 
concerning  the  real,  up  to  date  status 
of  a  programmer’s  activity.  The  informa¬ 
tion  contained  in  the  UDF  constitutes 
the  kind  of  tangible  evidence  of  job 
status  needed  to  maintain  effective  man¬ 
agement  control  of  the  software. 

The  programmer  responsible  for  design  of 
a  CPC  begins  by  identifying  each  element 
or  subroutine  needed  to  complete  the 
CPC.  Each  CPC  is  named,  and  the  pro¬ 
grammer  then  prepares  a  UDF  for  each, 
with  a  cover  sheet  (or  equivalent)  which 
sets  a  schedule  for  the  completion  of 
design,  coding,  checkout,  and  technical 
documentation.  As  each  of  these  detailed 
tasks  is  completed,  the  results  are 
placed  in  the  UDF  along  with  the  date 
actually  completed  and  review  signatures 
are  entered  on  the  UDF  cover  sheet. 
Figure  4.1-1  contains  an  example  of  a 
typical  UDF  cover  sheet.  Examples  of  the 
results  which  may  be  included  in  a  UDF 
are:  text,  flowcharts  and  subroutine 
lists  for  completion  of  design;  current 
listings  of  main  program  and  subroutine 
compilations  and  load  maps  for  com¬ 
pletion  of  coding;  written  verification 
of  program  performance,  written 
conformation  of  operational  functioning, 
output  verification  for  technical 
performance  and  copies  of  draft  and 
final  documentation  for  technical 
documentations. 
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CPC  # 


(Title) 


Completed 


Reviewed 


Descriptive  Text  Complete 

Flowcharts  Complete 

Subroutines  Identified 

Program  Coded  and  Compiled 

Program  Loaded  with  Dummy  Subroutines 

Program  Debugged  with  Dummy  Subrout ines_ 

Program  Loaded  with  Subroutines 

Functional  Interaction  Verified 

All  Operational  Modes  Functioning 

Output  Verified 

Draft  Documentation  Complete 

Final  Documentatn  ,i  Complete 


Figure  4.1-1.  Example  of  UDF  Cover  Sheet 
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The  UDF  then  becomes  the  mechanism 
whereby  a  programmer  can  easily  review 
his  task  schedule,  report  completion  of 
tasks,  and  record  acceptance  of  results. 
The  degree  to  which  the  UDF  is  complete 
provides  the  best  visibility  of  the 
"percent"  of  completion.  It  establishes 
naming  and  referencing  conventions  for 
the  products  of  CPCs  which  make  these 
products  more  accessible  to  other 
project  personnel.  Communication  between 
the  individual  assigned  a  particular 
series  of  CPCs  and  his  superior  is  more 
efficient  (and  less  costly)  since 
tangible  results  are  available  for 
objective  and  pertinent  review.  Using 
the  information  from  the  cover  sheet 
from  the  UDFs,  a  software  development 
progress  report  for  each  CPCI  can  be 
generated  as  in  Figure  4.1-2.  The  con¬ 
tractor  should  have  a  form  of  the  pro¬ 
gress  report  tailored  to  his  require¬ 
ments.  Air  Force  representatives  must 
know  the  content  of  the  progress  report 
used  since  it  might  be  displayed  during 
a  program  review  as  discussed  in 
paragraph  4.2. 

Looking  at  the  Development  Progress 
Report,  there  are  three  percentages  dis¬ 
played.  The  first  is  the  %  category  in 
the  far  left  hand  column.  This  value  is 
an  estimate  of  percent  of  the  total 
effort  each  CPC  requires.  With  this  per¬ 
centage,  a  manager  is  provided  a  numeri¬ 
cal  representation  of  the  importance  of 
each  CPC. 

The  second  percentage  appears  across  the 
report  opposite  the  CPC/Module  Name. 
This  is  an  estimate  of  the  percentage  of 
the  effort  for  a  CPC  that  the  associated 
development  requirement  takes.  These 
values  will  vary  depending  upon  the  pro¬ 
ject  specifications.  The  development 
requirement  percentages  result  from  coor¬ 
dination  by  the  Air  Force  acquisition 
engineer  and  the  contractor.  Some  of  the 
considerations  which  influence  these 
values  are  the  size  and  complexity  of 
the  CPCs,  the  number  and  size  of  the 
subroutines  for  each  CPCI  and  the  number 
of  new  concepts  to  be  developed 
requiring  more  design  development  than 
normal.  The  last  percentage  is  the  % 


work  complete  in  the  last  column  on  the 
right.  As  each  requirement  for  the  CPC 
is  completed,  it  is  approved  by 
management  on  the  UDF  and  checked  off  on 
the  development  progress  report.  Figure 
4.1-2  contains  an  example  of  a  CPC 
titled  DEMOPLBK  and  indicates  it  is 
completed  through  PROGRAM  LOADED  WITH 
SUBROUTINES  signified  by  the  checks  in 
appropriate  columns.  Taking  the  per¬ 
centages  from  the  top  of  each  column  and 
totaling  them,  the  value  of  percent 
work  complete  in  this  example  is  55%. 
Thus  the  progress  report  shows  visually 
each  CPC,  its  importance,  and  complete¬ 
ness  for  review  of  the  program  manager 
and,  if  desired,  by  the  USAF  acquisition 
engi neer. 

During  development  from  program  start 
through  PDR  and  up  to  C DR,  the  various 
CPCIs  are  defined  with  descriptive  text 
and  flowcharts  designed  for  identified 
CPCs.  These  products  are  very  difficult 
to  define  in  terms  of  partial  completion 
due  to  variables  of  size  and  importance 
changing  during  development.  Therefore, 
only  full  completion  is  normally  used  to 
accurately  determine  their  status. 

In  measuring  coding  and  checkout, 
several  functions  are  involved.  Some  of 
these  include  program  coding,  compiling, 
loading  with  dummy  subroutines, 
debugging  with  dummy  subroutines,  and 
finally  loading  with  the  functional  sub¬ 
routines.  Additionally,  if  a  large 
number  of  subroutines  are  required, 
coding  and  compiling  of  these  is 
required.  One  way  to  estimate  partial 
coding  completion  is  to  compare  the 
lines  of  code  to  the  flowchart.  Using 
the  percent  of  the  flowchart  completed 
as  a  guideline,  an  estimate  of  completed 
code  can  be  determined  on  a  one  to  one 
basis.  If  one  fourth  of  the  flowchart  is 
coded,  it  may  be  assumed  one  fourth  of 
the  coding  is  complete. 

4.1.2  Technical  Performance  Monitoring 

This  process  involves  measuring  and  eval¬ 
uating  the  degree  to  which  the  evolving 
ATE/TS  software  meets  the  requirements 
established  for  it.  The  primary  require- 
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Figure  4. 1-2  Example  of  Development  Progress  Report 
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ments  with  which  the  USAF  acquisition 
engineer  should  be  concerned  are  those 
reflected  in  the  Required  Operational 
Capability  (ROC),  Request  for  Proposal 
(RFP),  contractor's  technical  proposal 
and,  beyond  all  else,  the  ATE/TS  system 
specification  and  CPCI  specifications. 
Methods  by  which  periodic  reviews  of 
technical  performance  is  monitored  is 
included  in  paragraph  4.2. 

The  primary  source  of  information  by 
which  technical  performance  is  monitored 
by  the  USAF  acquisition  engineer  is  con¬ 
tractor  supplied  status  information  in 
the  form  of  documentation,  data  and  test 
results.  Technical  performance  para¬ 
meters  are  included  in  paragraph 
4. 1.2. 2. 

4. 1.2.1  Milestone  Schedule  Status  Para¬ 
meters.  Having  established  a  framework 
and  methodology  for  maintaining  the 
schedule  baseline  for  a  software  develop¬ 
ment  project,  a  procedure  must  be  devel¬ 
oped  for  tracking  milestone  schedule 
status  against  committed  schedule  dates. 

Milestone  schedule  status  is  the  condi¬ 
tion  of  a  milestone  in  regard  to  actual 
performance  toward  its  scheduled  disposi¬ 
tion.  One  of  the  most  difficult  concepts 
to  communicate  relative  to  the  sched¬ 
uling/schedule  status  function,  is  the 
difference  between  a  schedule  date  and  a 
status  date  (just  as  there  is  a  differ¬ 
ence  between  a  committed  schedule  date 
and  a.  planning  or  target  date).  Each  is 
governed  by  a  separate  set  of  rules  and 
conventions.  The  two  dates  may  be,  and 
frequently  are,  equal  to  one  another  (in 
an  "on  schedule"  condition,  that  is), 
but  they  do  not  have  the  same  meaning. 

People  tend  to  think  in  terms  of  each 
scheduled  milestone  having  but  one  date 
and  that  the  latest  estimated  completion 
date  (status  date)  is  a  "recovery  sched¬ 
ule"  date  that  permits  the  original 
schedule  date  to  be  completely  ignored. 
Project  managers  will  save  themselves 
considerable  trouble  if  they  ensure  that 
their  project  personnel  have  a  common 


understanding  of  the  scheduling/schedule 
status  terms  and  definitions  in  use  on 
their  particular  project. 

Milestone  schedule  status  is  expressed 
in  terms  of  two  components:  (1)  a  con¬ 
dition  and  (2)  the  date  on  which  the 
milestone  condition  was  or  will  be 
attained.  Status  conditions  are: 

a.  Estimated  completion  (E) 

b.  Potential  slippage  (P) 

c.  Slippage  (S) 

d.  Accomplished  (A) 

e.  Cancelled  (X) 

f.  Suspended  (Y) 

g.  Deleted  (D) 

(1)  Estimated  completion  (E) 

(a)  The  anticipated  date  for 
completion  of  a  milestone  as  provided  by 
the  responsible  organization. 

(b)  This  E-date  must  be  in  the 

future  relative  to  the  current  status 
cutoff  date,  otherwise  it  is  invalid. 
The  status  cutoff  date  (or  status-to-be- 
reported-as-of  date)  is  the  calender 

date  through  which  status  must  be 

reported  during  the  current  reporting 

cycl  e. 

(c)  E-dates  are  provided  by  the 
responsible  organization  based  upon  its 
capability  to  perform  and  do  not  require 
negotiation  with  or  the  approval  of 
other  parties. 

(d)  To  establish  an  E-date  as  a 

new  schedule  date,  negotiation  between 
affected  parties  is  required  in  the 

schedule. 

(e)  If  the  reported  status  date 
is  equal  to  the  schedule  date,  and  the 
reported  status  condition  is  "E,"  the 
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milestone  is  in  an  on-schedule  condition 
and  retains  the  status  condition  of  "E" 
for  performance  reporting  purposes. 

(f)  This  condition  can  exist 
only  when  both  the  schedule  date  and  the 
estimated  completion  date  are  in  the 
future  with  respect  to  the  status  cutoff 
date;  for  example,  where  the  status  cut¬ 
off  date  is  07-08-78  and  both  the  sched¬ 
ule  and  estimated  completion  dates  are 
07-11-78. 

(g)  If  the  status  (estimated 
completion)  date  is  earlier  than  the 
schedule  date,  and  both  are  in  the 
future  with  respect  to  the  status  cutoff 
date,  the  milestone  is  in  an  "ahead  of 
schedule"  condition,  and  the  milestone 
retains  the  status  condition  of  "E."  An 
example  of  an  ahead-of-schedule  condi¬ 
tion  would  be  one  where  the  status  cut¬ 
off  date  is  07-08-78,  the  schedule  date 
is  07-15-78,  and  the  estimated  comple¬ 
tion  status  date  is  07-11-78. 

(2)  Potential  Slippage  (P) 

(a)  If  the  reported  estimated 
completion  date  is  later  than  the  sched¬ 
ule  date,  and  both  are  in  the  future 
with  respect  to  the  status  cutoff  date, 
the  milestone  is  in  a  potential  slippage 
condition,  and  the  reported  status 
condition  of  "E"  is  converted  to  "P"  for 
performance  reporting  purposes. 

(b)  As  an  example  of  a  P-condi- 
tion,  assume  that  the  status  cutoff  date 
is  07-08-78,  the  schedule  date  is 
07-15-78,  and  the  reported  estimated  com¬ 
pletion  status  date  is  07-22-78. 

(c)  Also,  if  a  schedule  date  is 
in  the  future  with  respect  to  the  status 
cutoff  date,  and  the  status  date  is  not 
applicable,  milestone  status  will  be  in 
a  P-condition. 

(3)  Slippage  (S) 

(a)  If  the  reported  estimated 
completion  status  date  is  later  than  the 
schedule  date,  and  the  schedule  date  is 
earlier  than  (prior  to)  the  status  cut¬ 


off  date,  the  milestone  is  in  an  S-condi- 
tion;  the  reported  status  condition  of 
"E"  is  then  converted  to  "S"  for  perfor¬ 
mance  reporting  purposes. 

(b)  An  example  of  an  S-condi- 
tion  is  where  the  status  cutoff  date  is 
07-08-78,  the  schedule  date  is  07-01-78, 
and  the  reported  estimated  completion 
status  date  is  07-15-78. 

(c)  Further,  if  a  schedule  date 
is  in  the  past  relative  to  the  status 
cutoff  date,  and  the  status  date  is  not 
applicable,  status  of  the  milestone  will 
be  in  an  S-condltlon. 

(4)  Accomplished  (A) 

(a)  When  a  reported  milestone 
accompl  ishment /comp let  ion/actual  has 
been  verified,  the  reported  accomplish¬ 
ment  date  and  status  code  of  "A"  for  the 
milestone  are  confirmed. 

(b)  The  A-date  is  verified  when 
the  established  output  product(s)  or 
officially  accepted  "surrogate"  evidence 
of  completion  (e.g.,  transmittal  letter, 
signed  and  approved  meeting  minutes, 
document  release  form  etc.)  is  confirmed 
to  be  available,  complete,  and  correct 
as  required.  Thus,  an  A-condition 
reported  by  a  responsible  organization 
must  be  verified  in  the  manner  defined 
for  that  particular  milestone. 

(5)  Cancelled  (X),  Suspended  (Y), 
or  Deleted  (D) 

(a)  Cancellation  (X)  action 
occurs  when  it  has  been  officially  deter¬ 
mined  that  a  once-valid  milestone 
requirement  no  longer  exists. 

(b)  Suspension  (Y)  action 
occurs  when  work  is  officially  stopped 
(held  in  abeyance),  usually  as  the 
result  of  USAF  direction. 

(c)  Deletion  (D)  action  occurs 
when  it  has  been  established  that  a  mile¬ 
stone  has  originated  in  error;  that  is, 
a  valid  milestone  requirement  never 
existed. 
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4. 1.2. 2  Technical  Performance  Param¬ 
eters.  The  parameters  associated  with 
measurement  of  the  technical  integrity 
of  ATE/TS  software  are  less  well  defined 
than  those  associated  with  cost,  sched¬ 
ule  and  documentation  status.  Here  the 
technical  proficiency  of  the  acquisition 
engineer  becomes  extremely  important.  In 
many  cases  technical  integrity  is  a 
matter  of  technical  judgement. 

In  TS,  for  example,  which  of  several 
numerical  integration  schemes  is 
selected  may  be  purely  a  matter  of  tech¬ 
nical  judgement.  However,  all  are  not 
equal  in  performance.  Therefore,  to  eval¬ 
uate  numerical  integration  schemes  appli¬ 
cable  in  any  given  situation  requires 
knowledge  of  calculus  of  finite  differ¬ 
ences  and  error  analysis.  If  called  upon 
to  exercise  judgement  in  technical 
disciplines  outside  his  skill  area,  the 
acquisition  engineer  should  seek 
consultant  assistance.  Beyond  this, 
however,  technical  integrity  of  ATE/TS 
software  can  be  evaluated  by  means  of 
the  following  parameters: 

a.  Specification  requirements  (Sec¬ 
tion  3.0).  The  contractor  is  obligated 
to  provide  a  system  meeting  these 
requirements.  Each  and  every  requirement 
contained  in  Section  3.0  of  the  ATE/TS 
system  and  CPCI  specifications  consti¬ 
tute  a  technical  evaluation  parameter. 

Paragraph  4.2  provides  examples  of  means 
by  which  contractors  evaluate  their  own 
performance  against  these  requirements. 
The  acquisition  engineer  should  insure 
that  reports  reflecting  performance 
against  each  and  every  such  requirement 
is  reported  on  a  regularly  occuring 
basis  and  that  the  form  of  the  informa¬ 
tion  is  satisfactory  for  his  evaluation 
purposes. 

b.  Specification  verification  require¬ 
ments  (Section  4.0).  These  constitute 
the  means  by  which  each  Section  3.0 
requirement  is  measured  in  order  to  veri¬ 
fy  that  the  ATE/TS  meets  this  require¬ 
ment.  Therefore,  each  Section  4.0  speci¬ 
fication  item  is  an  important  technical 


measurement  parameter.  Test  reports 
analysis  documents  and  inspection 
reports  provided  by  the  contractor  are 
analyzed  by  the  acquisition  engineer  to 
ensure  whether  the  required  verification 
has  been  demonstrated. 

c.  Specification  Applicable  Document 
requirements  (Section  2.0).  This  specifi¬ 
cation  section  defines  the  MIL-STDS, 
etc.,  which  are  applicable  to  the  ATE/TS 
system.  Therefore  requirements  included 
in  the  corresponding  MIL-STDS  and  other 
forms  of  command  media  constitute 
technical  performance  parameters. 

4.1.3  Documentation  Status 

As  the  entire  development  proceeds,  many 
documents  are  generated  and  accumulated 
for  inclusion  in  the  UDF.  The  USAF 
acquisition  engineer  should  require  for¬ 
mal  release  of  the  required  documents  as 
they  are  available.  As  discussed  in  the 
Guidebook  Computer  Program  Documentation 
Requirements,  the  items  listed  in  Figure 
4.1-3  are  the  documents  needed  for 
proper  USAF  utilization  of  the  software. 
These  are  the  documents,  the  status  of 
which,  the  acquisition  engineer  is 
primarily  concerned  with.  The  means  by 
which  this  status  is  obtained  is 
included  in  paragraph  4.2. 

The  Program  Manager  is  charged  with 
tracking  the  status  and  progress  of  docu¬ 
mentation.  It  is  his  responsibility  to 
confirm  that  the  documents  produced  will 
meet  quality  standards  on  which  he  and 
the  Air  Force  have  concurred,  in  order 
to  satisfy  the  needs.  Since  the  Program 
Manager  is  operating  in  a  cost 
controlled  environment,  and  document 
production  is  a  significant  cost  item, 
he  must  have  visibility  of  the  progress 
of  various  documentation  activities,  to 
effectively  control  costs  and  achieve 
his  profit  objectives. 

The  document  standards  employed  in  soft¬ 
ware  development  projects  have  as  their 
objective  the  production  of  pertinent 
documentation,  i.e.,  documents  tailored 
to  the  needs  of  the  Air  Force  and  pro- 
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Figure  4. 1-3.  Documentation  Needs  in  the  Computer  Program  Development  Cycle 


duced  early  enough  in  the  development 
activity  to  provide  necessary  visibility 
(particularly  to  the  USAF).  This  implies 
that  documents  must  be  planned  to  be  pro¬ 
duced  i in  parallel  with  other  development 
tasks  (usually  in  parallel  with  design 
activities),  rather  than  planned  as  the 
last  obligation  of  the  software  devel¬ 
oper.  Document  standards  are  developed 
and  applied  which  directly  address  the 
functional  needs  of  specifically-identi¬ 
fied  audiences.  Furthermore,  documents 
are  submitted  in  draft  form  to  the  USAF 
acquisition  engineer  for  early  review. 

The  standards  described  here  are  not  uni¬ 
versal;  rather,  they  are  project  unique, 
shaped  by  contractual  obligations,  USAF 
needs,  and  project  objectives.  The  speci¬ 
fic  document  plans  and  standards 
employed  may  vary  from  project  to  pro¬ 
ject,  but  they  exhibit  the  general  char¬ 
acteristics  discussed  herein.  These  new 
document  standards  may  somewhat  increase 
the  costs  associated  with  review  and 
packaging.  The  materials  are  now  more 
critically  reviewed  to  ensure  that  they 
meet  the  USAF's  needs,  and  multiple  docu¬ 
ments  are  being  packaged  for  publication 
instead  of  a  single,  omnibus  Maintenance 
Document.  However,  having  a  specific 
definition  of  form  and  content  of  a  docu¬ 
ment,  and  an  understanding  of  the  docu¬ 
ment's  intended  use,  the  programmer 
should  find  the  preparation  task  less 
burdensome  and  time-consuming.  Further, 
key  information  such  as  input/output  for¬ 
mats,  design  specifications  for  com¬ 
ponents  and  sub-routines,  and  standard 
nomenclature  conventions  is  recorded  and 
made  easily  available  to  allow  the  pro¬ 
grammer  and  integrator  greater  visi¬ 
bility  of  how  the  individual  portions  of 
a  software  capability  relate  to  each 
other.  This  visibility  should  help  each 
individual  to  perform  his  assigned  task 
with  increased  efficiency  and  less  cost. 

4.2  STATUS  REVIEW 

This  section  describes  the  review  pro¬ 
cesses  required  by  USAF  command  media, 
both  for  contractor  and  government  use, 
and  those  in  use  primarily  by  contrac¬ 


tors  which  have  arisen  as  a  result  of 
experience  with  a  number  of  ground  sys¬ 
tems. 

4.2.1  Contractor  Internal  Reviews 

The  purpose  of  contractor  internal 
reviews  is  to  assess  on  a  continuing 
basis  the  ATE  or  TS  software  in  terms  of 
task  results.  The  results  reviewed 
include  plans,  documents,  software  ele¬ 
ments,  data  and  supporting  materials.  In 
addition  these  reviews  are  conducted  to 
assess  performance,  including  quality  of 
the  software  and  of  the  personnel 
actions  associated  with  it. 

Internal  review  conducted  by  contractors 
is  a  continuing  process.  While  the  speci¬ 
fics  vary  between  contractors  normally 
the  techniques  employed  haVe  the  char¬ 
acteristics  described  below. 

4. 2. 1.1  MIC  Facility.  A  physical  room, 
referred  to  herein  as  the  Management 
Informantion  Center  (MIC),  is  set  aside 
in  which  computer  program  status  informa¬ 
tion  is  permanently  displayed.  Normally 
one  or  more  individuals  are  assigned  the 
responsibility  to  collect  periodic  sta¬ 
tus  information  from  the  software  man¬ 
ager  and  maintain  the  MIC  information  in 
a  current  status.  The  contractor  organi¬ 
zation  performing  this  activity  is  nor¬ 
mally  the  Program  Planning  and  Control 
(PP&C)  function.  However,  it  is  funda¬ 
mental  to  the  management  philosophy  asso¬ 
ciated  with  MIC  activities  that,  while 
PP&C  may  physically  collect  and  display 
software  status  information,  the  ATE  or 
TS  software  manager  is  responsible  for 
the  accuracy  and  validity  of  information 
displayed  therein.  For  this  reason  this 
manager  normally  acknowledges  the  infor¬ 
mation  by  placing  his  signature  on  the 
displayed  information  after  PP&C  has  pre¬ 
pared/reviewed  the  displays. 

4. 2. 1.2  MIC  Function.  The  information 
displayed  in  the  MIC  concerns  all  impor¬ 
tant  attributes  of  the  status  of  the  TS 
or  ATE  and  is  normally  updated  weekly. 
Figure  4.2-1  contains  an  example  of  MIC 
display  material  associated  with  the 


software  test  status  for  a  typical  TS. 
Here  the  parameter  measured,  and  status 
displayed,  is  the  number  of  software 
certification  tests  successfully 
completed  versus  the  number  scheduled. 

In  general  the  contractor  will  select, 
for  MIC  display,  information  depicting 
status  of  each  and  every  performance 
parameter  which  the  contractor  is  obli¬ 
gated,  by  virtue  of  his  contract,  to 
demonstrate  as  a  condition  of  USAF 
acceptance  of  the  TS  or  ATE  system. 

The  program  manager  is  obligated,  by 
virtue  of  the  ATE  or  TS  system  specifi¬ 
cation,  and  each  software  configuration 
item  (CPCI)  specification,  to  verify  per¬ 
formance.  These  parameters,  therefore, 
constitute  his  contractual  obligation. 
Figure  4.2-2  contains  a  hypothetical  MIC 
room  display  example  for  a  training  sim¬ 
ulator  whose  partial  requirements  are  as 
follows: 

a.  A  separate  configuration  item 
exists  for  the  TS  computer  software  and 
the  Computer  Generated  Imagery  (CGI) 
system  software. 

b.  The  specifications  require  that 
the  CGI  software  occupy  not  more  than 
50%  of  the  central  processing  unit  (CPU) 
capacity  and  not  more  than  75%  (24. 4K) 
of  the  main  storage  capacity.  It  further 
requires  that  not  more  than  4.65  Mega 
bytes  of  disk  storage  capacity  are  to  be 
used. 

c.  The  specifications  require  that 
the  TS  software  use  not  more  than  90%  of 
the  CPU  and  not  more  than  30. 3K  words  of 
the  32K  main  storage  capacity. 

Evidently  the  following  events  are 
reflected  on  the  TS  Main  Memory  Utiliza¬ 
tion  status  chart.  Figure  4.2-2. 

a.  The  requirements,  as  reflected  in 
the  applicable  specification,  changed  as 
a  result  of  two  Engineering  Change  Pro¬ 
posals  (ECP).  While  the  original  specifi¬ 
cation  allowed  the  contractor  to  use 
only  29. 7K  words  of  main  memory  this  was 


relaxed  to  30. IK  as  a  result  of  ECP  270 
and  further  relaxed  to  30. 7K  by  ECP  320. 

b.  The  contractor,  during  the  period 
starting  early  1976  and  continuing  for 
more  than  a  year  had  considerable  diffi¬ 
culty  meeting  this  criteria.  However,  as 
a  result  of  corrective  software  redesign 
actions  taken  in  mid  1977  he  was  able  to 
meet  the  requirement  and,  as  of  the  date 
of  this  information,  the  TS  software  per¬ 
formance  meets  its  memory  utilization 
design  capacity. 

Note,  also,  on  Figure  4.2-2,  that  the 
specification  verification  method  is  evi¬ 
dently  different  for  the  two  computer 
systems.  While  the  TS  computer  capacity 
must  be  verified  by  actual  measurement 
in  some  prescribed  form,  the  CGI  soft¬ 
ware  is  verified  by  analysis. 

Figure  4.2-3  provides  hypothetical 
examples  of  other  parameters  which  may 
be  displayed  in  a  MIC  environment.  These 
charts  depict  the  status  of  coding  and 
module  verification  (by  test)  of  changes 
being  incorporated  in  a  set  of  TS  test 
software. 

In  general  there  are  an  almost  endless 
variety  of  chart  and  display  formats 
which  appear  in  a  MIC  display.  However, 
the  general  requirements  are  summarized 
as  follows: 

a.  The  display  should  depict  the 
important  (specification  requirement) 
parameters  simply  and  consistently.  This 
aids  understanding  and  comprehension  of 
the  displayed  data. 

b.  It  should  be  possible  to  visually 
scan  the  information  and  determine 
whether  the  status  is  "good"  or  "bad" 
without  requiring  a  detailed  knowledge 
of  the  specification  requirement  or  of 
the  software  CPCI.  This  makes  it  possi¬ 
ble  for  contractor  and  government  per¬ 
sonnel  of  widely  varying  backgrounds  to 
understand  the  information. 

c.  The  charts  should  be  sufficiently 
simple  that  they  can  be  readily  compre- 
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hended  in  reduced  size.  The  examples 
given  were  taken  from  actual  MIC  boards, 
each  containing  two  or  more  charts, 
which  are  approximately  3  ft  x  4  ft  in 
size.  Yet,  the  information  reduced  to 
the  8  1/2  x  11  inch  format  of  the  fig¬ 
ures  is  still  legible. 

This  makes  it  possible  for  the  contrac¬ 
tor  to  distribute  MIC  information  to 
individuals  not  having  convenient  access 
to  the  MIC  room.  Further,  it  is  a  con¬ 
venient  form  for  distribution  of  MIC  con¬ 
tents  to  the  USAF  Systems  Program  Office 
(SPO). 

4. 2. 1.3  MIC  Advantages.  Normally  the 
MIC  is  available  for  review  by  all  TS  or 
ATE  program  key  personnel.  Therein, 
fully  visible,  is  displayed  all  problems 
so  that  coordinated  action  by  responsi¬ 
ble  managers  is  directed.  In  most 
successful  programs  the  MIC  room  is  a 
work  room  where  program  staff  meetings 
are  held  and  where  problems  are  dis¬ 
cussed,  actions  assigned  and  performance 
reviewed. 

In  actual  practice  on  most  programs  the 
MIC  room,  if  properly  developed  and  main¬ 
tained  is  occupied  by  the  Program  Man¬ 
ager  and  his  staff  a  very  significant 
portion  of  the  time.  It  is,  in  a  very 
real  sense,  the  control  center  of  a  con¬ 
tractor's  program  activities. 

4. 2. 1.4  MIC  Alternatives.  Although  the 
MIC  is  a  contractor  function,  and  is 
used  by  contractors  wh;*her  or  not  a  MIC 
is  specified  in  his  contract,  it  is 
recommended  that  ATE  or  TS  acquisition 
contracts  be  written  in  such  a  way  that 
the  contractor  is  obligated  to  provide 
facsimile  copies  of  all  MIC  information 
to  the  SPO  on  a  periodic  (i.e. ,  monthly) 
basis. 

In  this  way  the  ATE  or  TS  acquisition 
engineer  has  the  best  and  most  up  to 
date  information  available  to  him  on  the 
important  aspects  of  the  evolving  soft¬ 
ware  and  hardware  system.  Further,  this 
information  provides  an  excel lant  source 


of  discussion  items  at  periodic  Program 
Management  Reviews  (PMR)  held  between 
the  contractor  and  the  LISAF. 

If  the  contractor  is  not  obligated  to 
maintain  a  MIC  and  has  no  intent  of  main¬ 
taining  one  for  his  own  use  then  it  is 
recommended  monthly  reports,  reflecting 
actual  status  vs  planned  status  as 
follows: 

a.  Status  concerning  and  deliveries/ 
submittals  indicated  in  paragraph  3.2, 
should  be  reported.  In  particular,  the 
acquisition  engineer  should  ensure  that 
the  contractor  is  required  to  monitor 
his  actual  vs  planned  performance,  and 
report  this  to  the  USAF,  for  every 
Section  3  requirements  item  and  every 
Section  4  verification  item  in  the 
ATE/TS  System  Specification.  In  addi¬ 
tion,  items  selected  by  the  acquisition 
engineer  from  CPCI  specifications  should 
also  be  reported. 

b.  Information  reported  should,  like 
those  examples  indicated  in  paragraph 

4.2.1  above,  contain  technical 
requirement  performance  as  will  schedule 
and  cost  performance,  since  all  three 
elements  are  necessary  to  the  successful 
software  acquisition  engineering 
management  job. 

c.  Specific  information  necessary  to 
support  USAF  requirements  as  reflected 
in  MI L-STD- 1 52 1A  (See  paragraph  4.2.2), 
should  be  required  of  the  contractor. 

4.2.2  Government  Reviews 

MIL-STD-1521A,  Technical  Reviews  and 
Audits  for  Systems,  Equipments  and  Com¬ 
puter  Programs  is  applicable  to  TS  and 
ATE. 

4. 2. 2.1  Review  Types.  MIL-STD-1521A 
specifies  seven  separate  government 
reviews  as  defined  below.  Normally  these 
reviews  are  held  at  contractor  facili¬ 
ties.  Since  the  contractor  and  SPO/ 
acquisition  engineer  are  geographically 
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separated,  the  contractor  should  be 
required  to  submit  review  material  well 
in  advance  of  the  review  meeting. 

The  acquisition  engineer's  role  in  these 
reviews  is  extremely  significant;  he  is 
the  government's  primary  representative, 
ensuring  the  ATE/TS  system,  when 
delivered,  meets  USAF  requirements.  To 
carry  out  this  function  adequately  he 
must  be  armed  with  knowledge  of  the  soft¬ 
ware  requirements  as  reflected  in  the 
Request  for  Proposal  (RFP ) ,  bidding 
contractor's  proposal  and  the 
specifications.  It  is  therefore 
recommended  that  the  acquisition  engi¬ 
neer  carefully  refamiliarize  himself 
with  these  documents  before  attending 
any  of  the  government  reviews. 

The  following  paragraphs  in  this  section 
are  based  on  MIL-STD-1521A  and  indicate 
items  which  the  acquisition  engineer 
should  review  and  evaluate  during  con¬ 
duct  of  these  meetings.  It  is 
recommended  the  acquisition  engineer, 
using  these  as  a  "checklist,"  compile  a 
list  of  primary  requirements  in  precise 
quantitative  or  qualitative  obtained 
from  the  RFP,  technical  proposal,  ROC, 
specifications  (if  available)  and  any 
other  available  information,  and  take 
this  list  with  him  to  government  review 
meetings. 

During  these  meetings  the  contractor’s 
job  is  to  present  information  concerning 
the  evolving  ATE/TS  or  other  ground  sys¬ 
tem.  The  government's  job  is  to  review 
and  evaluate  the  information  and  then 
make  a  judgement  concerning  the  degree 
to  which  the  government's  requirements 
are  being  met.  The  list  of  requirements 
should  be  used  to  evaluate,  point  by 
point,  the  contractor's  ATE/TS  software 
system.  During  this  evaluation  the  con¬ 
tractor  should  be  required  to  discuss, 
or  demonstrate,  how  the  evolving  soft¬ 
ware  meets  each  requirement  contained  in 
the  list. 

As  a  word  of  caution,  it  should  be 
pointed  out  the  acquisition  engineer  is 
frequently  at  a  disadvantage  at  these 
meetings.  This  stems  from  the  fact  that 


the  contractor's  role  places  him  In  the 
position  of  controlling  the  information 
to  be  presented.  Secondly,  since  the  con¬ 
tractor  is  designing  the  system,  he  nor¬ 
mally  has  greater  knowledge  of  his 
design  than  the  acquisition  engineer. 

This  inherent  disadvantage  can  be  offset 
by  two  things.  First,  as  previously  Indi¬ 
cated,  the  acquisition  engineer  should 
come  with  his  own  information  and  knowl¬ 
edge  by  the  method  discussed  above. 
Secondly,  he  should  persist  <n  his 
efforts  to  require  the  contractor  to 
demonstrate  evidence  that  his  system 
design  meets  each  and  every  requirement. 
The  contractor  should  be  persuaded  to 
describe  not  only  how  his  system  works, 
but  hew  his  system  meets  the  require¬ 
ments. 

a.  System  Requirements  Review  (SRR). 
The  objective  of  this  review  is  to 
ascertain  the  adequacy  of  the  contrac¬ 
tor's  efforts  in  defining  system  require¬ 
ments.  It  is  conducted  when  a  signifi¬ 
cant  portion  of  the  system  functional 
requirements  has  been  established. 

b.  System  Design  Review  (SDR).  This 
review  is  conducted  to  evaluate  the  opti¬ 
mization,  correlation,  completeness,  and 
risks  associated  with  the  allocated  tech¬ 
nical  requirements.  Also  included  is  a 
summary  review  of  the  system  engineering 
process  which  produced  the  allocated 
technical  requirements  and  of  the  engi¬ 
neering  planning  for  the  next  phase  of 
effort.  This  review  is  conducted  when 
the  system  definition  effort  has  pro¬ 
ceeded  to  the  point  where  system  char¬ 
acteristics  are  defined  and  the  Cl  are 
identified. 

c.  Preliminary  Design  Review  (PDR). 
This  review  is  conducted  for  each  CPC  I 
or  aggregate  of  CPCIs  to  (1)  evaluate 
the  progress,  technical  adequacy,  and 
risk  solution  (on  a  technical,  cost,  and 
schedule  basis)  of  the  selected  design 
approach,  (2)  determine  its  compati¬ 
bility  with  performance  and  software 
requirements  of  the  CPCI  development 
specification,  and  (3)  establish  the 
existence  and  compatibility  of  the  physi- 
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cal  and  functional  interfaces  among  the 
CPCIs  and  items  of  equipment,  facilities 
and  personnel. 

d.  Critical  Oesign  Review  (CDR).  This 
review  is  conducted  for  each  Cl  when 
detail  design  is  essentially  complete. 
The  purpose  of  this  review  is  to  (1) 
determine  that  the  detail  design  of  the 
Cl  under  review  satisfies  the  perfor¬ 
mance  and  engineering  requirements  of 
the  Cl  development  specifications,  (2) 
establish  the  detail  design  compati¬ 
bility  among  the  Cl  and  other  CIs,  facil¬ 
ities,  computer  programs  and  personnel, 

(3)  access  producibi 1 ity  and  Cl  risk 
areas  (on  a  technical,  cost,  and  sched¬ 
ule  basis),  and  (4)  review  the  prelimi¬ 
nary  specifications. 

e.  Functional  Configuration  Audit 
(FCA).  A  formal  audit  to  validate  that 
the  development  of  a  Cl  has  been  com¬ 
pleted  satisfactorily  and  that  the  Cl 
has  achieved  the  performance  and  func¬ 
tional  characteristics  specified  in  the 
functional  or  allocated  identification. 

f.  Physical  Configuration  Audit 
(PCA).  A  technical  examination  of  a 
designated  Cl  to  verify  that  the  Cl  "As 
Built"  conforms  to  the  technical  docu¬ 
mentation  which  defines  the  Cl. 

g.  Formal  Qualification  Review  (FQR). 
The  test,  inspection,  or  analytical  pro¬ 
cess  by  which  products  at  the  end  item 
or  critical  item  level  are  verified  to 
have  met  specific  procuring  activity  con- 
tracual  performance  requirements  (speci¬ 
fications  or  equivalent).  This  review 
does  not  apply  to  requirements  verified 
at  FCA. 

4. 2. 2. 2  Review  Functions.  Figure  4.2-4 
illustrates  the  life  cycle  events  asso¬ 
ciated  with  acquisition  of  ATE  and  TS 
software  and  indicates  the  points  within 
this  life  cycle  where  formal  government 
reviews  take  place.  In  addition,  it 
indicates  topics  which  should  be 
reviewed  by  the  SPO  acquisition  engi¬ 
neering  staff  at  these  formal  mile¬ 
stones.  Topics  to  be  discussed  at  these 
reviews  are  indicated  below. 


a.  The  SRR  is  normally  conducted 
during  the  system  conceptual  or 
validation  phase.  Such  reviews  may  be 
conducted  at  any  time  but  normally  are 
conducted  after  the  accomplishment  of 
functional  analysis  and  preliminary 
requirements  allocation  to  ATE/TS 
computer  program  CIs  to  determine 
initial  direction  and  progress  of  the 
contractor's  system  engineering 
management  effort  and  his  convergence 
upon  an  optimum  and  complete  configura¬ 
tion.  Since,  for  these  systems  software 
cannot  really  be  separated  from  hard¬ 
ware,  both  disciplines  should  be 
reviewed.  The  total  system  engineering 
management  activity  and  its  output 
should  be  reviewed  for  responsiveness  to 
the  SOW  and  TS/ATE  system  requirements. 
Representative  items  to  be  reviewed 
should  include  the  results  of  the 
following: 

(1)  Requirements  Specification 
(See  the  Requirements  Specification 
Guidebook ). 

(2)  Functional  Flow  Analysis, 
including  total  ATE/TS  software  func¬ 
tional  flow  diagrams. 

(3)  System/Cost  Effectiveness 

Analysis. 

(4)  Trade  Studies  (e.g.  addressing 
ATE/TS  system  functions  in  hardware/ 
firmware/software ) . 

(5)  System  Interface  Studies,  such 
as  the  CGI  interfacing  with  the  motion 
system  for  a  TS  or  the  contractor 
furnished  software,  adaptors,  etc.  with 
purchased  ATE  hardware/software. 

(6)  Generation  of  Specifications. 

(7)  Program  Risk  Analysis. 

(8)  Integrated  Test  Planning. 

(9)  Producibility  Analysis  Plans 

(10)  Technical  Performance  Measure¬ 
ment  Planning 
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(11)  Engineering  Integration 

(12)  Data  Management  Plans 

(13)  Configuration  Management 

Plans 


(14)  Human  Factors  Analysis 

(15)  Life  cycle  cost  analysis 


The  contractor  should  describe 
gress  and  indicate  problems  in: 

his 

pro¬ 

(1)  Risk 
ranking. 

identification 

and 

risk 

(2)  Risk 

avoidance/reducti on 

and 

control . 

(3)  Significant  trade-offs  between 
ATE/TS  system  or  system  segment  specifi¬ 
cation  requirements/constraints  and 
resulting  engineering  design  require¬ 
ments/constraints  and  loyistic/cost  of 
ownership  requi rements/constrai  nts  and 
unit  production  cost/design-to-cost 
object  ? ves. 

(4)  Significant  hazard  considera¬ 
tion  should  be  made  here  to  develop 
requirements  and  constraints  to 
eliminate  or  control  these  system 
associated  hazards.  While  the  ATE/TS 
software  is  not  normally  involved  in 
system  safety  analyses  it  can  be  used  to 
improve  system  safety  problems. 

(5)  Information  which  the  con¬ 
tractor  identifies  as  being  useful  to 
his  analysis  and  available  through  the 
procuring  activity  should  be  requested 
prior  to  or  at  this  review  (e.g.,  prior 
studies,  operational/support  factors, 
cost  factors,  test  plan(s),  etc.).  A 
separate  SRR  may  be  conducted  for  each 
of  the  software  CIs  depending  upon  the 
nature  and  complexity  of  the  ATE/TS. 

After  completing  the  SRR,  the  contractor 
publishes  and  distributes  copies  of 
review  minutes.  The  procuring  activity 
officially  acknowledges  completion  of 
the  SRR. 


b.  The  SDR  is  conducted  to  evaluate 
the  optimization,  traceability,  corre¬ 
lation,  completeness,  and  the  risk  of 
the  allocated  requirements,  including 
the  corresponding  test  requirements  in 
fulfilling  the  ATE/TS  system  or  system 
segment  requirements  (the  functional 
configuration  baseline).  The  review 
encompasses  the  total  system  require¬ 
ments  as  well  as  the  ATE/TS  software.  A 
technical  understanding  is  reached  on 
the  validity  and  the  degree  of 
completeness  of  the  following  infor¬ 
mation: 

(1)  System  Specification 

(2)  CPCI  Specifications 

(3)  The  engineering/cost  of  the 

system 

An  SDR  is  conducted  for  ATE/TS  systems 
which  are  sufficiently  complex  to 
warrant  the  formal  assessment  of  the 
allocated  requirements  (and  the  basis  of 
these  requirements)  before  proceeding 
with  the  preliminary  design  of  CIs.  The 
SDR  is  primarily  concerned  with  the 
overall  review  of  the  operational/ 
support  requirements,  updated/completed 
system  specification  requirements,  allo¬ 
cated  performance  requirements,  and  the 
accomplishment  of  the  system  engineering 
management  activities.  The  purposes  of 
the  SDR  are  to: 

(1)  Insure  that  the 

updated/completed  system  specification 
is  adequate  and  cost  effective  in 
satisfying  USAF  requirements. 

(2)  Insure  that  the  allocated  soft¬ 
ware  requirements  represent  a  complete 
and  optimal  synthesis  of  the  system 
requi rements. 

(3)  Insure  that  the  technical 
risks  are  identified,  ranked,  avoided, 
and  reduced. 

(4)  Ensure  that  a  technical  under¬ 
standing  of  requirements  has  been 
reached  and  technical  direction  is  pro¬ 
vided  to  the  contractor. 
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The  SDR  Includes  a  review  of  the 
following: 

(1)  ATE/TS  Requirements  Analysis 

(2)  Functional  Analysis 

(3)  Requirements  Allocation 

(4)  System/Cost  Effectiveness 

(5)  Reliability/Maintainability 

(RAM) 

(6)  Electromagnetic  Compatibility, 
as  appropriate 

(7)  Software  Maintenance  Concept 

(8)  System  Safety 

(9)  Security 

(10)  Human  Factors 

(11)  Transportability  (including 
Packaging  and  Handling) 

(12)  Standardization 

(13)  Value  Engineering 

(14)  System  Growth  Capability 

(15)  Program  Risk  Analysis 

(16)  Technical  Performance  Measure¬ 
ment  Planning 

(17)  Producibility  Analysis  (i.e., 
significant  aspects  of  materials, 
tooling,  processes,  facilities,  skills, 
etc. ) 

(18)  Life  Cycle  Costing 

(19)  Computer  Program  Development 

Plan 

(20)  Design-to-Cost  Goals 

(21)  Environmental  Conditions  as 
these  apply  to  software  requirements 
(Temperature,  Vibration,  Shock, 
Humidity,  etc.) 


(22)  Results  of  significant  trade 
studies. 

Review  Section  4.0  of  the  ATE/TS  system 
specification  and  all  available  software 
specifications  for  format,  content,  tech¬ 
nical  adequacy,  and  completeness.  All 
available  test  documentation,  including 
Cl/subsystem  and  system  test  plans,  are 
reviewed  to  insure  that  the  proposed 
test  program  satisfies  the  test  require¬ 
ments  of  Section  4.0  of  the  system  and 
Part  I  Cl  development  specifications. 
All  entries  labeled  "not  applicable 
(N/A)"  or  "to  be  determined  (TBD)"  in 
Section  4.0  of  the  system  specification 
and  Part  I  Cl  development  specification 
are  identified  and  explained  by  the  con¬ 
tractor. 

The  following  topics  should  be  presented 
and  reviewed  for  the  software: 

(1)  An  overall  review  of  system 
requirements  to  assure  that  a  technical 
understanding  of  requirements  has  been 
reached. 

(2)  The  management  controls  and 
methodology  that  will  be  used  to  ensure 
satisfactory  design  of  computer  pro¬ 
grams,  including  the  techniques  to  pro¬ 
vide  traceability  of  requirements  from 
the  system  specification  through  the  com¬ 
puter  program  development  specifications 
and  continuing  through  the  computer  pro¬ 
gram  product  specifications. 

(3)  Identification  of  all  CPCIs 
required  throughout  the  system.  Examples 
are:  operational  programs;  maintenance/ 
diagnostic  programs;  test/debug  pro¬ 
grams;  exercise  and  analysis  programs; 
simulation  programs,  and  compilers/ 
assemblers  and/or  store  certification 
programs  and  other  required  support  pro¬ 
grams  (e.g.  bootstrap  loaders  and  other 
tools). 

(4)  The  schedule  for  the 
development  of  each  CPC  I  and  the 
procedure  for  monitoring  and  reporting 
status. 
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(5)  The  procedure  for  monitoring 
and  reporting  computer  program  sizing 
and  timing  data  and  data  base  storage 
requirements. 

(6)  Computer  programming  standards 
and  conventions  to  be  enforced  by  the 
contractor. 

(7)  Trade-off  and  design  studies 
that  have  applicability  for  decisions 
relating  to: 

(a)  data  base  design 

(b)  computer  program  language 
usage 

(c)  space  allocation 

(d)  operating  system  and/or 
executive  design 

(e)  computer  instruction  set 
selection 

(8)  The  computer  programming 
techniques  to  be  adopted  for  use  in  the 
system,  e.g.,  on-line  processing, 
off-line  processing,  parallel  or  multi¬ 
processing,  multi -programming,  time 
sharing,  etc. 

(9)  A  general  description  of  the 
size  and  operating  characteristics  of 
all  computer  programs  (e.g.,  operational 
programs,  maintenance/diagnostic  pro¬ 
grams,  compilers,  etc.)  to  include  data 
base  requirements. 

(10)  A  description  of  requirements 
for  system  exercising  and  identification 
of  functional  requirements  (exercise  con¬ 
figuration,  conditions,  frequencies, 
functional  simulation,  recording,  and 
analysis),  and  identification  of  major 
elements  required  to  implement  the  exer¬ 
cising  capabi lity. 

(11)  Identification  of  all  com¬ 
puter  programming  languages  to  be 
utilized  in  the  system,  and  a 
description  of  hew  each  language  impacts 
the  development,  and  the  operations, 
maintenance  and  test  areas. 


(12)  Identification  of  *'.e 
computer  facilities  needed  to  sup  ‘t 
computer  programs  in  the  deplo  .nt 
phase,  and  the  extent  to  which  ese 
facilities  will  be  provided. 

(13)  Review  of  data  interfaces 
with  existing  automatic  data  processing 
systems. 

After  completing  the  SDR,  the  contractor 
submits  copies  of  review  minutes.  The 
procuring  activity  officially  acknowl¬ 
edges  completion  of  the  SDR. 

c.  The  PDR  is  a  formal  technical 
review  of  the  basic  design  approach  for 
a  Cl  or  for  a  functionally  related  group 
of  CIs.  It  is  normally  held  after 
authentication  of  the  Part  I  development 
specification(s)  and  the  accomplishment 
of  preliminary  design  efforts,  but  prior 
to  start  of  the  detail  design. 

The  contractor  presents  a  review  of  the 
following: 

(1)  Preliminary  design  of  the 
ATE/TS  including  its  software. 

(2)  Trade-studies  and  design 
studies  results. 

(3)  Interface  requirements  con¬ 
tained  in  Part  I  development  specifica¬ 
tions  and  interface  control  data  (e.g., 
interface  control  drawings)  derived  from 
these  requirements. 

(4)  CPCI  development  schedule 

(5)  Value  Engineering  Considera¬ 
tions,  Preliminary  Value  Engineering 
Change  Proposals  (VECP)  and  VECPs  (if 
appl icable) 

(6)  Description  and  character¬ 
istics  of  "off-the-shelf"  hardware/ 
software  including  any  optional 
capabilities  such  as  special  features, 
interface  units,  special  instructions, 
controls,  formats,  etc. 
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(7)  Existing  documentation 

(technical  orders  (T.O.),  commercial 
manuals,  etc.)  for  "off-the-shelf" 
hardware/software  and  copies  of  con¬ 
tractor  specifications  used  to  procure 
equipment  are  made  available  for  review 
by  the  procuring  activity. 

(8)  Life  cycle  costing  analysis 

CPCI  and  other  non-hardware  considera¬ 
tions: 

(1)  The  computer  program  func¬ 
tional  flow  should  be  completed  to  a 
flow  chart  level  which  allocates  the 
Part  I  performance  and  design  processing 
requirements  to  the  individual  computer 
program  components  of  the  CPCI. 

(2)  Storage  Allocation  Charts. 
This  information  is  detailed  for  each 
CPCI  as  a  whole,  describing  the  manner 
in  which  available  storage  is  allocated 
to  individual  Computer  Program 
Components  (CPCs).  Timing,  sequencing 
requirements,  and  relevant  equipment 
constraints  used  in  determining  the 
allocation  are  included. 

(3)  Control  Functions  Description. 
A  description  of  the  executive  control 
and  start/recovery  features  for  the 
computer  program  system  should  be 
available,  including  method  of 
initiating  system  operation  and  features 
enabling  recovery  from  system 
malfunction. 

(4)  Program  Structure.  The 
contractor  describes  the  overall 
hierarchial  structure  of  the  computer 
program,  the  reasons  for  choosing  the 
software  modules  described,  the  extent 
to  which  top-down  development  will  be 
used  within  the  constraints  of  the 
available  computer  resources,  and  any 
support  programs  which  will  be  required 
in  order  to  develop/maintain  the  program 
structure  and  allocation  of  data 
storage. 

(5)  Security.  An  identification  of 
unique  security  requirements  and  a 
description  of  the  techniques  to  be  used 


for  implementing  and  maintaining 
security  within  the  data  processing  sub¬ 
system  shall  be  provided. 

(6)  Reentrancy.  An  identification 
of  any  reentrancy  requirements  and  a 
description  of  the  techniques  for  imple¬ 
menting  reentrant  routines. 

(7)  Computer  Program  Development 
Facility.  The  availability,  adequacy, 
and  planned  utilization  of  the  computer 
program  development  facility  should  be 
addressed. 

(8)  Computer  Program  Development 

Facility/Operational  System.  The  con¬ 
tractor  provides  information  relative  to 
unique  design  features  which  may  exist 
in  a  computer  program  component  in  order 
to  allow  use  within  the  computer  program 
development  facility,  but  which  will  not 
exist  in  the  ATE/TS  to  be  delivered.  The 
contractor  should  provide  information  on 
the  design  of  support  programs  not 
explicitly  required  for  the  ATE/TS 

system  but  which  will  be  generated  to 
assist  software  development. 

(9)  Development  Tools.  The 

contractor  should  identify  any  special 
simulation,  data  reduction  or  utility 
tools  that  are  not  deliverable  under  the 
terms  of  the  contract,  but  which  are 
planned  for  use  during  program 

development. 

(10)  Review  word  lengths,  message 

formats,  storage  available  within  the 
computer,  card  and  magnetic  tape/disk 
formats,  timing,  and  other  considera¬ 
tions  which  were  established  in  the  Part 
I  CPCI  development  specification.  At 

this  time,  the  interfaces  between  CPCI 
and  hardware  CIs  shall  be  defined 
sufficiently  to  enable  computer  program 
design  to  proceed  independently. 

(11)  Analyze  word  formats,  card 
and  magnetic  tape/disk  formats,  transfer 
rates,  etc.  for  incompatibilities.  For 
interfaces  with  other  CIs  or  CPCIs  which 


are  the  responsibility  of  another  con¬ 
tractor  or  government  agency,  draft  and/ 
or  final  Interface  Control  Drawings 
(ICDs)  should  be  reviewed. 

(12)  Review  all  functional 
interfaces  between  CPCIs  within  the 
system.  (A  more  detailed  review  of  these 
interfaces  at  a  lower  level  is  conducted 
at  the  CDR  or  at  an  In-Progress  Review 
prior  to  CDR). 

(13)  Review  the  structure  of  the 
CPCI  as  a  whole  with  emphasis  on  the 
following: 

(a)  Allocation  of  computer 
program  components  to  functions 

(b)  Storage  requirements  and 
allocation 

(c)  Computer  program  operating 

sequences 

(d)  Design  of  the  data  base 

(14)  Analyze  critical  timing 
requirements  of  the  system  as  they  apply 
to  the  CPCI  to  insure  that  proposed  CPCI 
design  will  satisfy  the  timing 
requirements.  Review  estimated  running 
time  given  by  the  contractor  for 
compatibility  with  timing  requirements. 

(15)  Review  the  CPCI  interactions 
with  the  human  factors  requirements. 
Review  the  man-machine  interfaces 
including  operator-inserted  on-line 
commands  (e.g.,  format  of  command  state¬ 
ments,  switch  actions),  formats  of 
machine-generated  alerts,  and  initial 
draft  of  display  and  hardcopy  output 
design. 

d.  The  CDR  is  conducted  on  each  Cl 
prior  to  start  of  coding  to  insure  that 
the  detail  solutions  as  reflected  in  the 
draft  Part  II  software  specification 
satisfy  performance  requirements 
established  by  the  Part  I  development 
specification. 

The  CDR  for  ATE/TS  sofware  is  a  formal 
technical  review  of  the  CPCI  detail 


design.  The  CDR  is  normally  accomplished 
for  the  purpose  of  establishing  integ¬ 
rity  of  computer  program  design  at  the 
level  of  flow  charts  or  computer  program 
logical  design  prior  to  coding  and 
testing.  For  less  complex  CPCIs,  the  CDR 
may  be  accomplished  at  a  single  review 
meeting.  The  primary  product  of  the  CDR 
is  formal  identification  of  specific  com¬ 
puter  programming  documentation  which 
will  be  released  for  coding  and  testing. 

Since  computer  program  development  is  an 
iterative  process,  the  completion  of  a 
CDR  for  a  CPCI  is  not  necessarily  suffi¬ 
cient  for  maintaining  adequate  visibil¬ 
ity  into  the  development  effort  through 
testing.  Additional  Technical  Inter¬ 
changes  (TI)  or  PMRs  may  be  scheduled 

post-CDR  which  address: 

(1)  Responses  to  outstanding 
action  items 

(2)  Modifications  to  design 

necessitated  by  approved  ECPs  of 
design/program  errors 

(3)  Updating  sizing  and  timing 

data 

(4)  Updated  design  information,  as 
applicable 

(5)  Development  status  reports 

(6)  Results  obtained  during 

in-house  testing,  including  problems 

encountered  and  solutions  implemented  or 
proposed. 

Items  to  be  reviewed.  The  contractor,  as 
a  minimum,  should  review  the  following: 

(1)  Draft  of  complete  Part  II  CPCI 
specification  with  exception  of  instruc¬ 
tion  listings,  etc.  which  can  only  be 
produced  after  coding  the  program.  In 
cases  where  the  CDR  is  conducted  in 
increments,  a  complete  draft  Part  II  may 
not  be  made  available  until  the  last  one 
is  conducted.  If  conducted  in  increments 
the  review  would  be  conducted  on  the 
detail  design  of  the  CPC(s)  being 
reviewed. 
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(2)  Supporting  documentation 

describing  results  of  analyses,  testing, 
etc.,  as  mutually  agreed  by  the  pro¬ 
curing  activity  and  the  contractor. 

(3)  System  Allocation  Document  for 
Cl  inclusion  at  each  scheduled  location. 

The  contractor  should  provide  informa¬ 
tion  on  firmware  which  is  included  in 
"off-the-shelf"  equipment  or  to  be 
included  in  equipment  developed  under 
the  contract.  Firmware  in  this  context 
includes  the  microprocessor  and  asso¬ 
ciated  sequence  of  micro-instructions 
necessary  to  perform  the  allocated 
tasks.  As  a  minimum,  the  information  pre¬ 
sented  during  CDR  shall  provide  descrip¬ 
tions  and  status  for  the  following: 

(1)  Detailed  logic  flow  diagrams 

(2)  Processing  algorithms 

(3)  Circuit  diagrams 

(4)  Block  and  timing  data  (e.g. , 
timing  charts  for  micro-instructions) 

(5)  Memory  (e.g.,  type  Read  Only 
Memory  (ROM),  Programmable  Read  Only 
Memory  (PROM)  word  length,  size  (total 
and  spare  capacity) 

(6)  Micro-instruction  list  and  for¬ 
mat. 

e.  The  objective  of  the  FCA  is  to 
verify  that  the  CPCIs  actual  performance 
complies  with  its  Part  I  development 
specification.  Test  data  is  reviewed  to 
verify  that  the  ATE/TS  software  has  per¬ 
formed  as  required  by  its  functional 
and/or  allocated  configuration  identifi¬ 
cation. 

The  FCA  for  a  complex  ATE/TS  may  be  con¬ 
ducted  on  a  progressive  basis.  The  FCA 
is  conducted  on  that  configuration  of 
the  Cl  which  is  representative  (proto¬ 
type  or  pre-production  article  is  not 
produced,  the  FCA  is  conducted  on  a 
first  production  article. 


Prior  to  the  FCA  data  (for  CPCIs  to  be 
audited),  the  contractor  should  provide 
the  following  information  to  the  pro¬ 
curing  activity. 

(1)  Contractor  representation  (the 
test  manager  should  be  in  attendance). 

(2)  Identification  of  software 
items  to  be  audited: 

(a)  CPCI  Identifications 

(b)  Specification  Identifica¬ 
tion 

(c)  Current  listing  of  all 
deviating/waivers  against  the  Cl,  either 
requested  of,  or  approved  by  the 
procuring  activity. 

(d)  Status  of  Test  Programs  to 
test  configured  items  with  ATE  (when 
applicable). 

The  contractor's  test  procedures  and 
results  are  reviewed  for  compliance  with 
specification  requirements. 

The  following  testing  information  should 
be  available  for  the  FCA  team. 

(1)  Test  plans/procedures  and 
available  acceptance  test  plans/during 
which  pre-acceptance  data  was  recorded. 

(2)  A  complete  list  of 
successfully  accomplished  functional 
tests  which  pre-acceptance  data  was 
recorded. 

(3)  A  complete  list  of  successful 
functional  tests  if  detailed  test  are 
not  recorded. 

(4)  A  complete  list  of  functional 
test  required  by  the  specification  but 
yet  not  performed.  (To  be  performed  as  a 
system  or  subsystem  test). 

Testing  accomplished  with  the  approved 
test  procedures  and  validated  data  (wit¬ 
nessed)  are  sufficient  to  insure  Cl 
performance  as  set  forth  in  the 
specification  Section  3  and  meet  the 
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quality  assurance  provisions  contained 
in  the  specification  Section  4.  For 
those  performance  parameters  which 
cannot  completely  be  verified  during 
testing,  adequate  analysis  or  simula¬ 
tions  should  have  been  accomplished.  The 
results  of  the  analysis  or  simulations 
are  used  to  insure  configuration  item 
performance  as  outlined  in  the  speci¬ 
fication. 

The  contractor  should  provide  the  FCA 
team  with  a  briefing  for  each  CPCI  being 
FCA'd  and  delineate  the  Cl/Subsystem 
test  results  and  findings  for  each  CPCI. 
The  discussion  should  include  require¬ 
ments  of  the  development  specification 
that  he  was  not  able  to  meet  including  a 
proposed  solution  to  each  item,  an 
account  of  the  ECPs  incorporated  and 
tested  as  well  as  proposed,  and  a 
general  presentation  of  the  entire  Cl 
development  test  effort  delineating  pro¬ 
blem  areas  as  well  as  accomplishments. 

An  audit  of  the  Cl/Subsystem  Preliminary 
Qualification  Test  (PQT)  and/or  Formal 
Qualification  Test  (FQT),  plans/proce¬ 
dures  should  be  made  and  compared 
against  the  official  test  data.  The 
results  are  checked  for  completeness, 
accuracy,  etc.  Deficiencies  are  docu¬ 
mented  and  made  a  part  of  the  FCA  min¬ 
utes.  Completion  dates  for  all  discrep¬ 
ancies  are  established  and  documented. 

An  audit  of  the  draft/final  Cl/Subsystem 
test  report  is  performed  to  validate 
that  the  report  is  accurate  and  com¬ 
pletely  describes  the  development  tests. 


After  completion  of  the  FCA,  the  contrac¬ 
tor  distributes  copies  of  the  FCA  min¬ 
utes.  The  procuring  activity  officially 
acknowledges  completion  of  the  FCA. 

f.  The  PCA  is  the  formal  examination 
of  the  as-built  version  of  a  Cl  against 
its  technical  documentation  in  order  to 
establish  the  CPCI’s  product  baseline. 
After  successful  completion  of  the 
audit,  all  subsequent  changes  are 
processed  by  ECP  action.  The  PCA  also 
determines  that  the  acceptance  testing 
requirements  prescribed  by  the  docu¬ 
mentation  is  adequate  for  acceptance  of 
production  units  of  a  CPCI  by  quality 
assurance  activities.  The  PCA  includes  a 
detailed  audit  of  specifications,  techni¬ 
cal  data  and  tests  utilized  in  develop¬ 
ment  of  the  software  and  a  detailed 
audit  of  technical  descriptions,  flow 
charts,  listings,  manual /handbooks  for 
CPCIs.  The  review  includes  an  audit  of 
the  released  software  documentation  and 
quality  control  records  to  make  sure  the 
as-built  configuration  is  reflected  by 
this  documentation. 

The  contractor  provides  the  following 
information  to  the  procuring  activity. 

(1)  Contractor  representation  (the 
test  manager,  as  well  as  software  man¬ 
ager,  should  be  in  attendance). 

(2)  Identification  of  items  to  be 
accepted  by: 

(a)  Specification  identifica¬ 
tion  number 


All  ECPs  that  have  occurred  during  the 
program  are  reviewed  to  assure  that  they 
have  been  technically  incorporated  and 
verified  during  the  development  test  pro¬ 
gram. 

PDR  and  CDR  minutes  should  be  examined 
to  assure  that  all  findings  have  been 
incorporated  and  completed. 

The  interface  requirements  and  the  test¬ 
ing  of  these  requirements  are  reviewed. 


(b)  Cl  identifiers 

(c)  Code  identification  numbers 

(d)  Computer  program  identifica¬ 
tion  numbers 


(e)  Computer  program  VDD 

(f)  Current  listings,  flow 
charts,  analyses  and  other  software 
supporting  documentation 
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(g)  TRDs  for  ATE  systems  and 
ATE  test  software 

(3)  A  list  delineating  all  devia¬ 
tions/waivers  against  the  CPCI,  either 
requested  or  procuring  activity 
approved. 

The  following  should  be  performed  by  the 
contractor  on  each  CPCI  being  PCA'd  and 
evaluated  by  the  acquisition  engineer: 

(1)  Review  Part  II  specification 
for  format  and  completeness 

(2)  Review  FCA  minutes  for 
recorded  discrepancies  that  required 
action 


(3)  Review  CPC  descriptions  and 
flow  charts 


(4)  Review  CPC 

requi rements 


interface 


(5)  Review  data 

characteristics,  storage 
charts  and  timing  and 
characteristics 


base 

allocation 

sequencing 


(6)  Review  flow  charts  for  proper 
entries,  symbols,  label  tags 

(7)  Compare  top-level  CPCI  flow 

charts  with  CPC  flow  charts 

(8)  Compare  detailed  CPC  flow 
charts  with  coded  program  for  accuracy 
and  completeness 


(9)  Check  positional  handbooks, 
computer  user's  manuals,  and  computer 
programming  manuals  for  format 
completeness  and  conformance  with 
applicable  data  items.  (Formal 
verification/acceptance  of  these 
handbooks/manuals  should  be  withheld 
until  system  testing  to  insure  that  the 
procedural  contents  are  correct. 


(10)  Examine  actual  Cl  (card 
decks,  tapes,  etc.,)  to  insure 
conformance  with  Section  5  of  the  Part 
II  specification 


(11)  Cross-check  a  current 
listings  of  instructions  with  the 
listing  in  the  Part  II  specification 

g.  The  objective  of  the  FQR  is  to 
verify  that  the  actual  performance  of  a 
CPCI  as  determined  through  test  complies 
with  its  Part  I  development 
specification,  and  to  identify  the  test 
report (s)/data  which  documents  results 
of  qualification  tests  of  the  Cl.  The 
point  of  government  certification  will 
be  determined  by  the  SPO  and  depends 
upon  the  nature  of  the  program,  risk 
aspects  of  the  particular  CPCI,  and 
contractor  progress  in  successfully 
verifying  the  Cl  requirements.  When 
feasible,  the  FQR  is  combined  with  the 
FCA  at  the  end  of  CPCI  subsystem 
testing,  prior  to  PCA.  If  sufficient 
test  results  are  not  available  at  the 
FCA  to  insure  the  CPCI  will  perform  in 
its  ATE/TS  environment,  the  FQR  is  con¬ 
ducted  (post  PCA)  during  system  testing 
whenever  the  necessary  tests  have  been 
successfully  completed  to  enable  CPCI 
certification.  For  non-combined  FCA/ 
FQRs,  traceability,  correlation,  and  com¬ 
pleteness  of  the  FQR  is  maintained  with 
the  FCA  and  duplication  of  effort 
avoided. 

4. 2. 2. 3  Other  Reviews.  In  addition  to 
those  formal  government  reviews 
indicated,  two  types  of  less  formal 
reviews  are  normally  held. 

a.  The  first  of  these  is  the  Program 
Management  Review  (PMR),  held  at  either 
the  government's  or  contractors  facili¬ 
ties.  Its  purpose  is  the  review  of  pro¬ 
gram  status  and  special  problems  by  key 
contractor  and  government  personnel  on  a 
regular  basis.  These  are  normally  held 
quarterly,  or  more  frequently  as  needs 
warrant,  and  the  dates,  locations  and 
agenda  are  directed  by  the  SPO.  The 
acquisition  engineer's  role  in  these 
reviews  is  to  assist  with  identification 
of  agenda  items  and  participating  in 
these  meetings  as  directed  by  the  SPO. 

b.  The  least  formal  government  review 
is  the  Technical  Interchange  (TI).  This 
varies  from  a  telephone  conversation 
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Section  5.0  PROBLEM  RECOGNITION  AND  CORRECTION 


We  have  come  now  to  the  most  important 
reason  for  accurate  and  complete  status 
measuring  and  reporting,  namely-recog¬ 
nizing  and  correcting  problems.  The  two 
prime  methods  of  recognizing  and 
correcting  software  development  problems 
are: 

a.  Schedule  Reviews  (Administrative) 

b.  Testing  (Technical) 

The  best  planned  and  adhered  to  sched¬ 
ules  will  not  insure  project  success  if 
the  product  design  is  deficient.  The 
project's  test  program  is  an  integral 
part  of  the  project's  master  schedule 
and,  it  is  the  prime  mechanism  for 
recognizing  technical  problems  and  high 
risk  areas. 

Management  techniques  for  recognizing 
problem  and  risk  areas,  categorizing  as 
to  impact  and  severity,  documenting  the 
problem,  planning  and  tracking  the  solu¬ 
tion,  are  discussed  in  this  section.  For¬ 
mally  documented  mechanisms  for 
elevating  problems  through  contractor 
management  hierarchy  are  described. 
Handling  routine  development  problems 
through  normal  change  processing 
channels  is  treated  in  the  guidebook  on 
Configuration  Management.  This  section 
of  this  book  concentrates  on  ways  in 
which  major  program  impact  problems  are 
recognized  and  averted. 

5.1  RECOGNIZING  HIGH  RISK  AREAS 


c.  Core  and  timing  utilizations 
exceeding  original  estimates 

d.  Changes  to  processor  architecture 

e.  Available  facilities  for  con¬ 
current  hardware  and  software  develop¬ 
ment 

f.  Complexity  of  operating  system 
software. 

Routine  "make  work"  type  of  design  pro¬ 
blems  are  an  inherent  element  in  any 
software  development  effort.  Most  pro¬ 
blems  in  this  category  are  remedied  by 
routine  handloads  or  "patches"  to  the 
baseline  program.  When  a  major  problem 
arises  requiring  a  major  program 
redesign  a  complete  recompilation  of 
the  program  may  be  required.  These  major 
recompilation  efforts,  if  unforeseen  and 
unscheduled,  can  cause  critical  program 
impact.  Overtime,  additional  manpower 
and  facilities  may  be  in  vain  in 
attaining  recovery.  Unforeseen  changes 
to  requirements,  processor  architecture, 
etc.  cannot,  in  general,  be  planned  for 
and,  as  a  result,  projects  will 
invariably  run  the  risk  of  slippages. 
Sometimes  problems  can  be  categorized  as 
"out-of-scope"  and  solutions  can  be 
renegotiated  including  new  schedules. 
Design  or  administrative  deficiencies 
which  necessitate  "in-scope"  changes  to 
maintain  spec  compliance  are  the  ones 
which  threaten  project  success  and  which 
we  shall  discuss  here. 


Hopefully  decisions  reached  early  in  the 
system  acquisition  phase  will  minimize 
high  risk  areas  of  software  development. 
Nevertheless,  some  high  risk  software 
development  problems  frequently  prevail 
such  as: 

a.  Faulty  or  late  availability  of 
support  software 

b.  New  or  frequent  changes  to  perfor¬ 
mance  and  interface  requirements 


5.1.1  Testing 

Verification  and  test  strategies  planned 
and  documented  in  the  CPDP  serve  to 
detect  major  design  deficiencies  as 
early  as  possible.  Well  planned  test 
philosophies  supplemented  by  accurate 
and  complete  status  reporting  result  in 
effective  problem  recognition.  Although 
testing  philosophy  is  more  thoroughly 
discussed  in  guidebooks  on  verification 
and  validation,  the  test  program  affects 
methods  of  reporting  status,  and  hence 
general  principles  are  discussed  here. 
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5.1.2  "Block"  Change  Testing 

One  approach  to  software  testing  which 
aids  in  early  detection  of  major  soft¬ 
ware  or  system  level  incompatibilities 
is  the  so  called  "Block  Change" 
approach.  (See  the  guidebook  on  Software 
Configuration  Management,  Section  5.0, 
for  a  thorough  discussion  of  this 
testing  strategy.)  In  this  approach 
software  is  developed  and  tested  in  pro¬ 
gressively  maturing  "blocks"  of  accumu¬ 
lated  changes  and  enhancements  to  the 
ultimate  CPCI.  The  initially  developed 
components  are  tested,  integrated  and 
tested  again  for  compatibility,  and  then 
validation  tested  as  an  initial  version 
of  the  CPCI. 

This  initially  validated  version  serves 
as  a  baseline  against  which  changes  can 
be  incorporated  in  order  to  support 
level  testing.  For  example,  for  TS  pro¬ 
grams,  initial  "Blocks"  are  made  avail¬ 
able  early  in  system  development  to  sup¬ 
port  hardware/software  compatibility 
testing.  As  higher  level  tests  and  other 
sources  of  performance  changes  (i.e., 
USAF  directed)  are  received  by  the  con¬ 
tractor  software  design  organization, 
they  can  either  be  held  open  until  the 
next  recompilation  of  the  program,  or 
they  can  be  incorporated  by  handload  or 
patch.  (Project  needs  dictate  which 
approach  is  used.)  When  the  volume  of 
such  "patches"  against  a  given  "Block" 
of  CPCI  versions  has  reached  unmanage¬ 
able  proportions  or  when  changes  are  too 
complex  to  "patch",  a  new  "Block"  or  pro¬ 
gram  recompilation  of  reworked  source 
code  is  produced.  This  creates  the  next 
"Block"  or  baseline  for  a  given  CPCI. 
The  Block  approach  is  primarily  designed 
to  ease  management  of  changes  and  to 
prioritize  development  of  CPCI  com¬ 
ponents  to  support  the  next  level  of  sys¬ 
tem  testing.  The  approach  provides,  as  a 
side  benefit,  an  "early  warning"  system 
for  discovering  major  performance 
deficiencies. 

In  summary,  therefore,  the  "Block" 
approach  serves  two  purposes.  First,  it 
simplifies  change  management  by  creating 


progressively  maturing  baselines  for 
each  CPCI  against  which  work  can  be 
prioritized  and  against  which  changes 
can  be  written  and  more  easily  managed. 
Second,  it  provides  early  recognition  of 
major  design  deficiencies  by  exercising 
initial  versions  of  the  end  item  in  a 
system  environment  early  in  the  develop¬ 
ment  phase  to  detect  problems  in  areas 
such  as  percentage  of  available  CPU 
time,  memory  utilization,  and  interface 
compatibility. 

This  approach  is  most  effective  on  TS 
software  development  because  of  the 
greater  intermodule  dependencies  and  the 
impact  of  program  directed  changes. 

The  same  approach,  on  a  smaller  scale, 
can  however  be  applied  to  ATE  software 
development,  particularly  to  initial 
integration  of  control,  support  and  test 
programs. 

5.1.3  Memory  and  Timing  Utilization 
Di agrams 

Another  technique  for  early  recognition 
of  high  risk  software  development  prob¬ 
lems  is  the  use  of  CPU  memory  and  timing 
utilization  diagrams.  These  diagrams 
help  the  p  )grammer  evaluate  his  core 
space  allocation  requirements  vs., 
remaining  available  memory.  Timing 
limitations  on  data  transfer  or  pro¬ 
cessing  operations  are  similarly  criti¬ 
cal  to  program  development. 

Figure  5.1-1  is  a  typical  memory  and 
timing  utilization  chart.  Use  of  these 
or  similar  techniques  provides  the  visi¬ 
bility  needed  to  track  code  development 
against  memory  and  timing  capacities. 
Failure  to  heed  these  limitations  may 
necessitate  major  redesigns. 

5.1.4  Requirements  Compliance 
Monitori ng 

One  of  the  most  effective  ways  of  recog¬ 
nizing  high  risk  technical  software 
development  problems  is  through  a  pro¬ 
gram  of  continuous  traceability  and  moni¬ 
toring  of  how  well  test  results  satisfy 


performance  requirements.  A  well 
designed  test  function  serves  this  objec¬ 
tive  by: 

a.  Providing  an  objective  evaluation 
of  performance 

b.  Providing  objective  evidence  of 
performance  verification  through  analy¬ 
sis  or  test 

c.  Providing  traceability  of  perfor¬ 
mance  and  test  requirements  and  changes 
to  actual  test  procedures  and  results 

d.  Providing  an  assessment  of  how 
well  the  requirement  was  satisfied  by 
the  analyses  or  tests  performed. 

There  are  typically  3  levels  of  testing 
required  to  fully  check  out  and  "sell 
off"  software: 

a.  Module  verification  test 

b.  Intermodule  tests 

c.  System  validation  tests 

Module  and  intermodule  tests  are 
so-called  "single  thread"  tests.  They 
individually  evaluate  a  given  set  of 
functional  requirements  allocated  to  mod¬ 
ules  or  interfaces.  For  example,  in  TS 
software,  a  module  designed  to  simulate 
the  function  "conventional  gravity 
weapon  ejector"  would  be  tested 
thoroughly  in  all  of  its  possible  modes 
or  configurations.  All  logic  branches 
would  be  executed  with  all  possible  com¬ 
binations  of  input  conditions  until  the 
module  functioned  correctly.  Similar 
tests  are  run  at  the  intermodule  level 
insuring  all  interfaces  between  modules 
are  exercised  with  all  (within  reason) 
probable  interface  conditions.  Finally 
the  integrated  program  is  "validation 
tested"  in  a  simulated  real  time  system 
environment  during  which  time  all  logic 
paths  may  or  may  not  be  exercised.  This 
final  validation  test,  however,  need  not 
execute  all  available  code  segments  if 
lower  level  tests  have  been  properly  com¬ 
pleted. 


Management  needs  visibility  over  all  3 
levels  of  testing  to  Insure  adequate 
fulfillment  of  requirements.  Status 
charts  designed  to  show  percentage  of 
tests  complete  per  module  together  with 
test  results  (test  reports  and  assess¬ 
ment  summaries)  are  necessary  measures 
of  software  development  progress.  These 
charts/summaries  should  be  reviewed  as 
part  of  periodic  schedule  performance 
reviews.  Equipped  with  these  parameters 
of  progress,  project  management  is  now 
in  a  position  to  recognize  and  abate 
major  schedule  impact  problems. 

5.2  PROBLEM  ABATEMENT  METHODS 

The  methods  to  be  employed  to  plan  and 
execute  problem  solutions  for  software 
vary  with  project  size  and  complexity. 
Problems  should  be  worked  through  a 
hierarchy  of  problem  reporting  and 
correction  mechanisms  as  a  function  of 
the  complexity,  cost,  schedule,  number 
of  parties  involved,  and  the  degree  of 
agreement  which  can  be  reached  among  the 
concerned  parties.  Figure  5.2-1  depicts 
the  a  trail  of  mechanisms  for  elevating 
a  problem  through  the  ranks  until  an 
acceptable  solution  is  reached.  The 
majority  of  ATE  and  TS  software  problems 
are  encountered  and  resolved  at  the  3 
lowest  levels  shown  in  this  figure.  Typi¬ 
cally,  when  a  problem  is  encountered  in 
design,  software  test  or  system  test,  a 
Software  Problem  Report  (SPR)  (see  the 
guidebook  on  Software  Quality  Assurance 
for  a  discussion  of  SPR  processing)  is 
processed.  A  contractor  engineering 
change  then  embodies  the  fix  in  the 
program  design  and  defines  the  retest 
requirements.  The  SPR  is  then  logged  and 
tracked  for  discrepancy  trends.  The 
foregoing  is  a  routine  which  results  in 
acceptable  solutions  to  the  vast 
majority  of  software  discrepancies. 

Problems  unsol vable  by  simple  SPR/ 
Committed  Class  II  change  are  elevated 
to  a  "formal  Action  Item."  These  are 
assigned  by  the  software  project  manager 
when  immediate  action  is  required  to  pre¬ 
clude  a  potential  "Area  of  Concern"  or 
"Top  Program  Problem"  (see  Figure  5.2-2) 
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from  developing.  Formal  Action  Items  are 
tracked  by  the  schedule  group  and 
require: 

a.  An  assigned  action  manager 

b.  Evidence  of  completion 

c.  Recommended  closure 

d.  Action  plan 

e.  Due  dates 

Occasionally  major  impact  problems  are 
encountered  which  require  top  management 
attention: 

a.  Major  change  in  processor  operat¬ 
ing  system  or  architecture 

b.  Insufficient  memory 

c.  Undefined  or  ambiguous  require¬ 
ments 

d.  Loss  of  critical  skills 

These  problems  cannot  be  solved  on  a 
simple  SPR  or  Formal  Action  Item.  They 
are  presented  to  the  appropriate  levels 
of  contractor  management  on  an  "Area  of 
Concern"  or  "Top  Program  Problem."  The 
criteria  for  an  "Area  of  Concern"  is 
that  the  problem  has  not  yet  impacted 
the  program  but  could  if  unresolved  by  a 
certain  date.  Areas  of  concern  are  pre¬ 
sented  on  view  foils  to  the  project  man¬ 
ager  and  his  staff  and  if  accepted  as 
legitimate,  they  are  given  a  control  num¬ 
ber  and  tracked  to  resolution.  Resolu¬ 
tion  involves  whatever  is  necessary  to 
solve  the  problem.  A  mini -network  may  be 
required  and  a  "back-up"  plan  together 
with  associated  schedules.  These  "Areas 
of  Concern"  are  reviewed  weekly  at  sched¬ 
ule  reviews,  are  statused  and  either  con¬ 
tinued,  suspended,  or  closed. 

Problems  which  cannot  be  resolved  by  any 
of  these  means  and  which  will  probably 
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impact  contractual  requirements,  costs, 
or  schedule  milestones,  are  elevated  to 
the  category  of  "Top  Program  Problems." 
Top  program  problems  are  given  top  man¬ 
agement  emphasis. 

Upon  acceptance  by  the  program  manager, 
the  schedule  maintenance  organization  is 
responsible  for  formal  tracking, 
statusing  and  graphic  displays.  These 
problem  abatement  mechanisms  are  treated 
with  the  same  degree  of  formality  and 
intensity  of  attention  as  Tier  I  sched¬ 
ules. 

In  addition  to  reviewing  and  statusing 
known  problems,  each  functional  manager 
is  required  to  canvass  his  organization 
at  least  monthly  for  candidate  "Areas  of 
Concern."  If  none  exists,  he  is  required 
to  sign  off  to  that  effect  on  a  monthly 
"Canvass  of  Problem  of  Identification" 
form.  This  action  forces  functional  man¬ 
agers  to  examine  their  operation  and 
encourages  the  reporting  of 
deliquencies. 

In  addition  to  all  of  the  foregoing, 
military  contractors  today  are  being 
forced  to  manage  quality  costs  with 
greater  formality.  Quality  costs  are 
costs  associated  with  inspection,  test, 
scrap,  rework,  repetitive  failure, 
warranty  claims,  etc.  MIL-STD-1520A 
(Corrective  Action  and  Disposition  Sys¬ 
tem  for  Nonconforming  Material)  is 
becoming  a  standard  requirement  on  new 
contracts.  This  standard  requires  the 
formal  documentation  of  quality  costs 
and  the  establishment  of  failure  pre¬ 
ventive  mechanisms  such  as  the 
"Corrective  Action  Board."  Since  soft¬ 
ware  is  a  deliverable  product  (DOD 
5000.29)  it  falls  within  the  purview  of 
MIL-STD-1520A.  Excessive  failures  in  any 
given  category  (i.e.  logic,  data  base, 
computation,  I/O)  deserve  attention  for 
corrective  action.  Problem  trend 
analyses  and  reports  designed  for  this 
purpose  are  an  integral  part  of  measur¬ 
ing  and  reporting  software  status. 
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Section  6.0  ATE  &  TS  MEASUREMENT  AND  REPORTING  VARIANTS 


There  are  numerous  variants  in  the  devel¬ 
opment  of  ATE  and  TS  software  which 
should  be  considered  in  the  early  stages 
of  system  acquisition  planning.  Consider¬ 
ation  of  these  variants  can  help  USAF 
and  contractor  management  plan  early 
warning  systems  for  avoiding  pitfalls 
which  can  seriously  impact  the  project, 
if  not  detected  and  remedied  early.  This 
section  describes  some  of  these  variants 
which  should  be  considered  during  ATE 
and  TS  software  schedule  and  performance 
status  reviews. 

6.1  ATE  VARIANTS 

Some  of  the  major  ATE  unique  considera¬ 
tions  which  should  be  examined  and 
planned  for  early  in  system  development 
are  described  below. 

6.1.1  Integration  of  Software 
Components 

Component -wise  ATE  software  is  a  compos¬ 
ite  of  vendor  and  contractor  developed 
programs.  The  control  and  support  pro¬ 
grams  are  usually  developed  by  the  ven¬ 
dor  well  in  advance  of  the  UUT  programs. 
When  finally  tested  as  an  integrated 
entity,  incompatibilities  are  frequently 
encountered  requiring  numerous  changes. 
Early  integration  of  UUT  programs  with 
vendor  software  help  detect  major  incom¬ 
patibilities  before  they  cause  insignifi¬ 
cant  impact. 

6.1.2  Change  Management. 

Change  management  is  generally  more 
difficult  for  ATE  programs  than  for  TS 
programs  simply  because  of  the  higher 
volume  and  larger  number  of  sources  of 
change.  Nearly  every  UUT  design  change 
will  precipitate  an  ATE  program  change, 
whereas  only  significant  performance 
changes  necessitate  TS  program  changes. 
Moreover  numerous  UUT  design  groups  both 
within  the  contractor's  organization  and 
his  subcontractors  contribute  to  UUT  pro¬ 
gram  designs  and  change.  This  aggravates 
the  software  configuration  control  prob¬ 
lem. 


For  example,  because  of  budget 
pressures,  the  Department  of  Defense, 
Joint  Services,  and  industry  are 
encouraging  commonality  among  deployed 
ground  systems  users.  In  the  ATE  world, 
configuration  commonality  is  a  means  to 
an  end;  namely  lower  life  cycle  costs. 
The  value  of  equipment  commonality  is 
significantly  evident  when  examining  the 
costs  incurred  in  acquiring  and 
maintaining  intermediate  level 
maintenance  capability  for  a  large 
number  of  bases  and  installations.  UUTs 
developed  by  literally  hundreds  of 
subcontractors  must  be  diagnosed,  fault 
isolated,  repaired  and  retested  within 
minimal  turnaround  times.  The  cost  of 
developing,  acquiring  and  maintaining 
hundreds  of  sets  of  UUT  peculiar  ATE 
systems  would  be  staggering.  To  reduce 
these  procurement  costs,  ATE  commonality 
must  be  attained  while  maximizing  the 
universality  (capability  to  test  large 
numbers  of  different  UUTs)  of  the  ATE. 
Commonality  serves  the  following 
objectives: 

a.  Minimization  of  development  costs 

b.  Minimization  of  procurement  costs 

c.  Transportability  of  like  configura¬ 
tions  among  installations 

d.  Minimal  maintenance  costs 

Preserving  this  commonality,  however, 
becomes  a  monumental  "headache”  if  users 
and  UUT  subcontractors  are  continuously 
changing  requirements. 

For  every  UUT  change  necessitating  an 
ATE  change,  all  like  configurations 
deployed  at  all  sites  must  be  retro¬ 
fitted  to  preserve  this  commonality. 
Attempts  to  increase  the  "universality" 
of  the  tester  complicates  preserving 
commonality  because  all  using  sites  must 
have  identical  configurations  even 
though  a  given  site  might  not  need  that 
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capability.  This  creates  horrendous  con¬ 
figuration  management  problems,  espe¬ 
cially  if  serious  design  deficiencies 
result  in  significant  post-  deployment 
changes. 

6.1.3  Design  for  Testability 

Management  visibility  systems  must 
assure  that  design  groups  and  subcon¬ 
tractors  "design  and  testability,"  maxi¬ 
mizing  use  of  Built-In-Test-Equipment 
(BITE)  to  ease  the  UUT  program  develop¬ 
ment  task.  Designing  for  testability  is 
usually  ignored  in  the  early  stages  of 
prime  system  acquisition.  This  aspect  of 
UUT  development,  usually  addressed  in 
procurement  specs  as  a  design  goal,  is, 
in  general,  not  statused  as  a  measurable 
design  parameter.  Self-test  subroutines 
designed  into  the  UUT  micro-code  can 
immensely  simplify  the  UUT  programming 
task  downstream  while  simultaneously 
enhancing  fleet  maintenance. 

6.1.4  ATE  Growth  Potential 

As  UUT  functional  capability  expands  in 
terms  of  more  functions  per  UUT  or  more 
UUTs,  ATE  software  capabilities  must  be 
similarly  expanded.  For  example,  extend¬ 
ing  the  range  of  an  air  vehicle  could 
involve  additional  modes  of  navigation, 
more  complex  operational  algorithms, 
faster  CPU's,  and,  perhaps  additional 
defensive  avionics  capability.  Unless 
this  potential  growth  is  anticipated  and 
planned  for  early  in  the  ATE  system 
acquisition  cycle,  the  required  growth 
potential  may  not  be  provided.  Memory 
and  timing  utilization  charts  such  as 
Figure  5.1-1  include,  therefore,  a 
"design  threshold"  which  should  not  be 
exceeded  to  allow  for  anticipated 
growth. 

6.1.5  Proprietary  Software 

While  both  ATE  and  TS  systems  will 
generally  embody  some  proprietary  soft¬ 
ware  (part  of  the  vendor's  operating  sys¬ 
tem),  its  impact  on  functionality  and 
maintenance  of  the  end  item  CPC I  is 
potentially  greater  for  ATE  than  for  TS. 


This  is  because  many  ATE  operating  sys¬ 
tems  today  embody  stimuli /measurement 
routines  within  the  vendor  supplied 
operating  system  itself.  The  contractor 
developed  UUT  program  in  ATLAS  merely 
calls  up  these  vendor  routines  assuming 
inputs  are  validly  calibrated  and 
applied  and  that  outputs  are  validly 
measured.  A  greater  percentage  of  the 
actual  test  process  is  executed  by  ven¬ 
dor  developed  code  than  by  contractor 
developed  code.  In  TS  software  the  opera¬ 
tional  program  performance  is  more  depen¬ 
dent  on  contractor  developed  code  than 
on  vendor  code.  Proprietary  software, 
therefore,  is  of  special  importance  to 
ATE  software  development  and  hence 
should  be  reported  and  considered  as  a 
potential  "risk"  area  because  of  the 
difficulty  in  getting  documentation 
necessary  for  organic  maintenance  capa¬ 
bility. 

6.2  TS  VARIANTS 

In  addition  to  the  previously  discussed 
differences  in  structure,  design,  and 
manageability  of  TS  from  that  of  ATE 
software,  the  following  considerations 
uniquely  apply  to  TS. 

6.2.1  Modularity  of  Design 

Trainer  software,  because  of  its  size 
and  complexity  (unlike  a  given  UUT  pro¬ 
gram)  is  not  a  one  man  effort.  TS  soft¬ 
ware  development  must  be  developed  in 
many  cases,  concurrent  with  hardware. 
This  among  other  reasons,  necessitates 
modularity  of  design  so  that  the  many 
designs  can  be  accomplished  on  schedule. 
This  demands  well  structured  and  docu¬ 
mented  inter-element  interfaces.  The 
principles  of  top  down  structured  pro¬ 
gramming  characterized  by  one-input, 
one-output  control  structures  greatly 
aid  in  software  manageability.  Such 
techniques  allow  so  called  "plug-in  com¬ 
patible"  software  designs  without  intro¬ 
ducing  new  connections  with  the  rest  of 
the  design.  Such  design  provides 
disciplined  use  of  the  "GO  TO"  statement 
which  if  uncontrolled  can  introduce  many 
program  looping  possibilities  which 
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makes  correctness  checking  and  main¬ 
tenance  difficult.  Management  enforce¬ 
ment  of  disciplined  rules  for  design 
modularity  and  structuredness  is  a  con¬ 
sideration  unique  to  TS  and  other  real 
time  software  development  and  statusing. 

6.2.2  HOL  vs  Assembly  Language 

The  question  of  HOL  vs.  assembly  lan¬ 
guage  is  also  of  Interest  in  modern  simu¬ 
lator  systems.  ATE  programs,  conversely, 
are  written  exclusively  in  ATLAS  or  some 
offshoot  thereof.  A  HOL  has  the  obvious 
advantage  of  maximizing  a  programmer's 
productivity  by  allowing  him  to  generate 
the  maximum  number  of  lines  of  machine 
code  in  a  given  time.  Balanced  against 
this  benefit  however  must  be  the  cost  of 
and  time  expended  by  a  compiler. 
Compilers,  however,  automatical ly  take 
care  of  problems  like  keeping  track  of 
multiple  nested  loop  variables  which 
must  share  the  same  index  register,  a 
problem  burdensome  to  the  assembly 
language  programmer.  Nevertheless, 
experienced  assembly  language  pro¬ 
grammers  may  be  able  to  save  core  space 
and  execution  time  by,  for  instance 
saving  the  index  register's  contents  in 
a  register  not  referenced  by  an  inner 
loop.  The  resulting  program  may  execute 
faster  and  occupy  less  core  than  equiva¬ 
lent  compiler  produced  code.  Studies 
have  confirmed  the  following: 

a.  Compiler  produced  code  can  be  from 
1.15  to  3  times  as  voluminous  as  an 
equivalent  assembly  language  program. 

b.  Execution  time  for  compiler  pro¬ 
duced  code  can  be  from  1.15  to  4  times 
longer  than  an  assembly  language  program 
for  the  same  algorithm. 

Therefore,  if  the  option  is  available, 
TS  software  project  management  needs  to 
insure  that  trades  are  conducted  early 
in  the  detail  design  phase  -  whether  to 
write  smaller  and  faster  programs  in 
assembly  language  while  incurring  a 
greater  programming  cost  or  whether  to 
employ  HOL  sacrificing  execution  time, 
core  space  and  compiler  expense.  Usually 
the  answer  to  this  trade  is  "quantity  of 


systems"  dependent.  Assembly  language 
tends  to  be  more  economical  as  the  num¬ 
ber  of  systems  deployed  increases.  If, 
for  example,  using  an  Interpretive  com¬ 
piler  requires  an  extra  memory  chip  that 
costs  $20.00  and  100  systems  are  to  be 
built,  then  the  contractor  can  spend  an 
additional  $2000  to  produce  the  same  pro¬ 
gram  In  assembly  language  not  even  con¬ 
sidering  the  saving  in  execution  time. 

Unlike  ATE,  therefore,  management  trades 
such  as  this  must  be  made  early  to 
insure  a  viable  and  economical  approach 
to  TS  software  design  is  undertaken. 

6.2.3  Incompatibility  of  TS  Components 

A  complete  system  simulator  is  composed 
of  a  basic  computer  central  core  from 
Vendor  A,  peripherals  from  Vendor  B  and 
operational  programs  written  by  the  con¬ 
tractor.  Incompatibilities  between  the 
CPU  and,  for  example,  a  terminal  are  not 
uncommon.  It  is  the  responsibility  of 
the  prime  contractor  to  integrate  all  TS 
components  and  resolve  these  problems 
early.  Again,  early  warning  reporting 
systems  are  vital  to  this  effort.  It  is 
not  meant  to  imply  that  ATE  systems  are 
free  of  component  incompatibility  prob¬ 
lems;  however,  for  a  TS  system  there  is 
a  greater  technical  challenge  for  the 
contractor  as  a  system  integrator 
primarily  because  of  the  greater  con¬ 
tractor  role  as  the  operational  software 
developer.  Hence  management  visibility 
systems  should  mandate  early  system  com¬ 
patibility  tests  and  appropriate  report¬ 
ing  mechanisms. 

6.2.4  Non-Standard  Design 

TS  software  design  is  less  adaptable  to 
standardization  of  design  than  ATE 
software.  For  example,  the  central  core 
of  an  ATE  system  can  be  designed  and 
built  independent  of  the  application.  A 
standard  stimuli /measurement  signal 
repertoire  is  available  "off-shelf"  from 
the  ATE  vendor.  The  ATE  central  core 
composed  of  CPU,  memory  Input/Output 
(1/0),  stimulus  generators/measurement 
devices  provide  and  measure  AC,  DC, 
pulse  generation,  wave  form  synthesis 
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over  a  somewhat  standard  range  of 
amplitudes  and  frequencies,  irrespective 
of  the  application.  TS  systems,  on  the 
other  hand  are  designed  around  a  given 
training  mission  where  the  unique  soft¬ 
ware  portion  is  a  greater  proportion  of 
the  total  trainer  software  package  than 


it  is  for  an  ATE  package.  Accordingly, 
more  management  emphasis  must  be  placed 
on  developmental  testing  in  anticipation 
of  a  higher  risk  factor  due  to  the 
unknowns  associated  with  non-standard 
and  unproven  software  systems. 
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Section  8.0  MATRIX  -  GUIDEBOOK  TOPICS  VS  GOVERNMENT  DOCUMENTS 


The  elements  of  Figure  8.0-1  correspond  corresponding  topic  i 
to  sections  in  the  guidebook  wherein  the  discussed. 
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Section  9.0  GLOSSARY  OF  TERMS 


Algorithm  -  A  set  of  rules  or  processes 
for  solving  a  problem  in  a  finite  number 
of  steps.  This  software  procedure  can  be 
presented  to  a  computer  precisely  and  in 
a  standard  form,  the  computer  then 
taking  the  algorithm's  course  of  action 
to  solve  the  problem. 

Computer  Program  -  A  series  of  instruc¬ 
tions  or  statements  in  a  form  acceptable 
to  computer  equipment,  designed  to  cause 
the  execution  of  an  operation  or  series 
of  operations.  Computer  programs  include 
such  items  as  operating  systems,  utility 
programs,  and  maintenance/diagnostic  pro¬ 
grams.  They  also  include  applications 
programs  such  as  payroll,  inventory, 
control,  operational  flight,  strategic, 
tactical,  automatic  test,  crew  simulator 
and  engineering  analysis  programs.  Com¬ 
puter  programs  may  be  either  machine 
dependent  or  machine  independent,  and 
may  be  general  purpose  in  nature  or  be 
designed  to  satisfy  the  requirements  of 
a  specialized  process  of  a  particular 
user. 

Computer  Program  Development  Cycle  -  The 
computer  program  development  cycle  con¬ 
sists  of  six  phases:  analysis,  design, 
coding  and  checkout,  test  and  integra¬ 
tion,  installation  and  operation  and 
support.  The  cycle  may  span  more  than 
one  system  acquisition  life  cycle  phase 
or  may  be  contained  in  any  one  phase. 
(AFR  800-14,  Volume  II) 

Contra  t  -  A  legally  enforceable  agree¬ 
ment  t etween  two  parties  (AF/Contractor, 
Contractor/subcontractor)  which 

describes  a  program  for  product  acquisi¬ 
tion.  The  contract  contains  the  System 
Specifications,  the  Statement  of  Work, 
(SOW),  the  Contract  Data  Requirements 
List  (CDRL),  and  the  Work  Breakdown 
Structure  (WBS). 

Computer  Program  Component  -  A  subset  of 
a  computer  program  configuration  item 
that  takes  the  form  of  a  module  sub¬ 
routine,  program  unit,  etc. 


Computer  Program  Configuration  Items  -  A 
computer  program  or  aggregate  of  related 
computer  programs  designated  for  config¬ 
uration  management.  A  CPCI  may  be  a 
punched  deck  of  cards,  paper  or  magnetic 
tape  or  other  media  containing  a 
sequence  of  instructions  and  data  in  a 
form  suitable  for  insertion  in  a  digital 
computer. 

Configuration  Item  -  An  aggregation 
which  satisfies  an  end  use  function  and 
is  designated  for  configuration  manage¬ 
ment. 

Configuration  Management  -  A  management 
discipline  applying  technical  and  admin¬ 
istrative  direction  and  surveillance  to: 

(a)  Identify  and  document  the  func¬ 
tional  and  physical  characteristics  of  a 
configuration  item; 

(b)  Control  changes  to  those  charac¬ 
teristics;  and 

(c)  Record  and  report  change  process¬ 
ing  and  implementation  status. 

Control  Software  -  Software  used  during 
execution  of  a  test  program  which  con¬ 
trols  the  nontesting  operations  of  the 
ATE.  This  software  is  used  to  execute  a 
test  procedure  but  does  not  contain  any 
of  the  stimuli  or  measurement  parameters 
used  in  testing  a  unit  under  test.  Where 
test  software  and  control  software  are 
combined  in  one  inseparable  program, 
that  program  will  be  treated  as  trst 
software.  (AFLC  66-37) 

Data  Base  -  A  collection  of  program 
code,  tables,  constants,  interface  ele¬ 
ments  and  other  data  essential  to  the 
operation  of  a  computer  program  or  soft¬ 
ware  subsystem. 

Estimating  Model  -  A  graphical  or  mathe¬ 
matical  representation  (of  a  specific 
work  task)  which  is  utilized  to  calcu¬ 
late  the  approximate  cost  to  develop 
and/or  produce  a  desired  product. 
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Life  Cycle  Analysis  -  An  analysis  of  a 
systems  total  cost  to  the  government 
over  its  full  life.  It  would  include  the 
cost  of  development,  production,  opera¬ 
tion,  support,  and  if  applicable, 
disposal. 

Logic  Flow  -  A  diagrammatic  representa¬ 
tion  of  the  logic  sequence  for  a  com¬ 
puter  program.  Logic  flows  may  take  the 
form  of  the  traditional  flow  charts  or 
in  some  other  form  such  as  a  program 
design  language. 

Milestone  -  An  event  which  occurs  at  a 
defined  time  point  in  a  schedule  of 
planned  activities. 

Module  -  A  portion  of  the  entire  soft¬ 
ware  system  that  normally  is  one  or  more 
programs  to  accomplish  a  specified  task. 

Organic  -  A  term  used  to  designate  a 
task  performed  by  the  Air  Force  rather 
than  a  contractor. 

Program  Design  Language  -  An  English- 
1  ike,  special ly  formatted,  textual 
language  describing  the  control  struc¬ 
ture,  logic  structure,  and  general 
organization  of  a  computer  program. 
Essential  features  of  a  program  design 
language  are: 

(a)  It  is  an  English-like  representa¬ 
tion  of  a  computer  procedure  that  is 
easy  to  read  and  comprehend. 

(b)  It  is  structured  in  the  sense 
that  it  utilizes  the  structured  pro¬ 
gramming  control  structures  and  indenta¬ 
tion  to  show  nested  logic. 

(c)  It  uses  full  words  or  phrases 
rather  than  the  graphic  symbols  used  in 
flow  charts  and  decision  tables. 

Program  Planning  and  Control  (PP&C)  -  A 
contractor  and/or  SPt)  organization  vhose 
responsibility  includes  development  and 
maintenance  of  schedules  and  other  man¬ 
agement  visibility  and  control  informa¬ 
tion. 


Quality  Assurance  -  A  planned  and  sys¬ 
tematic  pattern  of  all  software  related 
actions  necessary  to  provide  adequate 
confidence  that  computer  program  config¬ 
uration  items  or  products  conform  to 
established  software  technical  require¬ 
ments  and  that  they  achieve  satisfactory 
performance. 

Software  -  A  combination  of  associated 
computer  programs  and  computer  data 
required  to  enable  the  computer  equip¬ 
ment  to  perform  computational  or  control 
functions. 

Source  Selection  -  The  process  of 
selecting  which  among  competing  contrac¬ 
tors  shall  be  awarded  a  contract.  A 
significant  portion  of  this  involves 
evaluation  of  proposals  to  determine  the 
degree  to  which  the  government's  require¬ 
ments  would  be  satisfied. 

Support  Software  -  Auxiliary  software 
used  to  aid  in  preparing,  analyzing  and 
maintaining  other  software.  Support  soft¬ 
ware  is  never  used  during  the  execution 
of  a  test  program  on  a  tester,  although 
it  may  be  resident  either  on-line  or  off¬ 
line.  Included  are  assemblies,  com¬ 
pilers,  translators,  loaders,  design 
aids,  test  aids,  etc.  (tflZ  66-37) 

System  Engineering  -  The  application  of 
scientific  and  engineering  efforts  to 
transform  an  operational  need  or  state¬ 
ment  of  deficiency  into  a  description  of 
systems  requirements  and  a  preferred  sys¬ 
tem  configuration  that  has  been 
optimized  from  a  life  cycle  viewpoint. 
The  process  has  three  principal  ele¬ 
ments:  functional  analysis,  synthesis, 
and  trade  studies  or  cost-effectiveness 
optimization. 

Test  Software  -  Programs  which  implement 
documented  test  requirements.  There  is  a 
separate  test  program  written  for  each 
distinct  configuration  of  unit  under 
test  (AFLC  66-37). 
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Validation  -  Computer  program  validation 
is  the  test  and  evaluation  of  the  com¬ 
plete  computer  program  aimed  at  ensuring 
compliance  with  the  performance  and 
design  criteria. 

Verification  -  Computer  program  veri¬ 
fication  is  the  iterative  process  of  con¬ 
tinuously  determining  whether  the  pro¬ 
duct  of  each  step  of  the  computer  pro¬ 
gram  acquisition  process  fulfills  all 
requirements  levied  by  the  previous 
step,  including  those  set  for  quality. 

Unit  Development  Folder  -  A  storage 
folder  maintained  by  a  computer  pro¬ 
grammer  where  essential  listings,  sched¬ 


ules,  documentation  and  data  are  main¬ 
tained  for  purposes  of  visibility  and 
management  approval. 

Work  Breakdown  Structure  -  A  standard 
method  Tor  structuring  a  program  i nto 
its  various  required  work  tasks.  A  Work 
Breakdown  Structure  Is  implemented  per 
MIL-STD-881A  under  the  guidance  In  AFR 
800-17.  When  subdivided  as  necessary  by 
the  contractor  to  identify  tasks 
associated  with  a  single  responsible 
organization,  it  provides  a  basis  for 
contract  planning,  status  determination, 
and  reporting. 
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Section  10.0  ABBREVIATIONS  AND  ACRONYMS 


ACWP 

Actual  Cost  of  Work  Perfor¬ 

CMP 

Configuration  Management  Plan 

mance 

CPC 

Computer  Program  Component 

AFAL 

Air  Force  Avionics  Lab 

CPC  I 

Computer  Program  Configuration 

AFLCP 

Air  Force  Logistics  Command 

Item 

Pamphlet 

CPDP 

Computer  Program  Development 

AFPRO 

Air  Force  Plant  Representative 
Office 

Plan 

CPR 

Cost  Performance  Report 

AFR 

Air  Force  Regulation 

CPU 

Central  Processing  Unit 

AFSCP 

Air  Force  Systems  Command 
Pamphlet 

CRISP 

Computer  Resources  Integrated 
Support  Plan 

ASPR 

Armed  Forces  Procurement 
Regulation 

CRM 

Contract  Responsibility  Matrix 

ASUPT 

Advances  Simulator  Under¬ 
graduate  Pilot  Training 

CRT 

Cathode  Ray  Tube 

C/SCSC 

Cost/Schedule  Control  Systems 

ATE 

Automatic  Test  Equipment 

Criteria 

ATLAS 

Abbreviated  Test  Language  for 
all  Systems 

C/SSR 

Cost/Schedule  Status  Report 

DBMS 

Data  Base  Management  System 

ATPG 

Automatic  Test  Program 

Generation 

D&D 

Design  and  Development 

BCWP 

Budgeted  Cost  of  Work 

Performed 

DID 

Data  Item  Description 

DOD 

Department  of  Defense 

BCWS 

Budgeted  Cost  of  Work 

Scheduled 

DODI 

Department  of  Defense 
Instruction 

BITE 

Built  in  Test  Equipment 

ECD 

Estimated  Completion  Date 

CCP 

Contract  Change  Proposal 

ECP 

Engineering  Change  Proposal 

CDR 

Critical  Design  Review 

ESD 

Electronic  Systems  Division 

CDRL 

Contract  Data  Requirements 

List 

FACI 

First  Article  Configuration 
Inspection 

CER 

Cost  Estimating  Relationship 

FCA 

Functional  Configuration  Audit 

CFSR 

Contract  Funds  Status  Report 

FQR 

Formal  Qualification  Review 

CGI 

Computer  Generated  Imagery 

FQT 

Formal  Qualification  Test 

Cl 

Configuration  Item 

FY 

Fiscal  Year 
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'  mCEDIHG  PMH  MOV  FILMED 
BLANK 


GRC 

General  Research  Corporation 

R&DA 

Requirements  Definition  and 
Analysis 

HEW 

Health,  Education  and  Welfare 

R&M 

Rel 1 abl 1 1 ty/Mal nt  al nabi 1 1 ty 

HOL 

Higher  Order  Language 

RCA 

Radio  Corporation  of  America 

IBM 

International  Business  Machine 

Company 

RDT&E 

Research,  Development,  Test 
and  Evaluation 

I  CD 

Interface  Control  Drawings 

RF 

Radio  Frequency 

IMS 

Integrated  Management  System 

RFP 

Request  for  Proposal 

I/O 

Input/Output 

ROC 

Required  Operational 

ITA 

Interface  Test  Adapter 

Capability 

LOE 

Level  of  Effort 

ROM 

Read  Only  Memory 

LRU 

Line  Replaceable  Unit 

RSS 

Regulations,  Specifications 
and  Standards 

LSI 

Large  Scale  Integrated 

Circuitry 

SAE 

Software  Acquisition 
Engineering 

MEAC 

Management  Estimate  at 

Completion 

SDC 

System  Development  Corporation 

MIC 

Management  Information  Center 

SIRD 

Software  Implementation 
Requirements  Document 

NAA 

North  American  Autonetics 

SOW 

Statement  of  Work 

0«M 

Operational  and  Maintenance 

SPO 

Systems  Program  Office 

0«S 

Operational  and  Support 

SDR 

System  Design  Review 

PAR 

Problem  Analysis  Report 

SPR 

Software  Problem  Report 

PCA 

Physical  Conf iguration  Audit 

SRR 

System  Requirements  Review 

PDR 

Preliminary  Design  Review 

STOL 

Short  Take  Off  and  Landing 

PIDS 

Prime  Item  Development 
Specifications 

TI 

Technical  Interchange 

PMP 

Program  Management  Plan 

T.O. 

Technical  Order 

PMR 

Program  Management  Review 

TRD 

Test  Requirement  Document 

PQT 

Preliminary  Qualification  Test 

TS 

Training  Simulator 

PP&C 

Program  Planning  and  Control 

UDF 

Unit  Development  Folder 

PROM 

Programmable  Read  Only  Memory 

UUT 

Unit  Under  Test 

VAR 


Variance  Analysis  Report 
Version  Description  Document 


Validation  and  Verification 


VDD 

VECP 


Value  Engineering  Change 
Proposal 


**e  * 

-  '  ' 
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V&V 

vv&c 

MBS 


Verification,  Validation  and 
Certification 

Work  Breakdown  Structure 


Section  11.0  SUBJECT  INDEX 


SUBJECT 

ATE  and  TS  Variants 
Control  Rooms 
Critical  Design  Reviews 
Formal  Qualification  Review 
Functional  Configuration  Audit 
Management  Review 
Multiple  Tier  Schedules 
Networks 

Physical  Configuration  Audit 
Preliminary  Design  Review 
Problem  Abatement 
Program  Management  Review 
Risk  Assessment 
Schedule  Control 
Schedule  Development 
Schedule  Planning 
Status  Monitoring 
Status  Parameters 
System  Design  Review 
System  Requirements  Review 
Technical  Interchange 
Testing 

TS  and  ATE  Overview 
Types  of  Milestones 
Unit  Development  Folder 
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